[Laszlo-checkins] r12963 - in openlaszlo/trunk/WEB-INF/lps: config server/src/org/openlaszlo/compiler server/src/org/openlaszlo/sc
Max Carlson
max at openlaszlo.org
Fri Feb 27 19:26:45 PST 2009
I'd suggest hashing in-memory javascript/as files to reduce memory usage
and string comparison time for subsequent recompiles.
hqm at openlaszlo.org wrote:
> Author: hqm
> Date: 2009-02-19 22:42:40 -0800 (Thu, 19 Feb 2009)
> New Revision: 12963
>
> Modified:
> openlaszlo/trunk/WEB-INF/lps/config/lps.properties
> openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/Parser.java
> openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/SWF9Writer.java
> openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/Compiler.java
> openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9External.java
> openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9Generator.java
> openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/ScriptCompilerInfo.java
> Log:
> Change 20090220-hqm-R by hqm at badtzmaru.home on 2009-02-20 01:18:46 EST
> in /Users/hqm/openlaszlo/trunk3
> for http://svn.openlaszlo.org/openlaszlo/trunk
>
> Summary: support for swf9 incremental compilation
>
> New Features:
>
> Bugs Fixed:
>
> Technical Reviewer: dda
> QA Reviewer: ptw
> Doc Reviewer: (pending)
>
> Documentation:
>
> Release Notes:
>
> Details:
>
> This change allows the use of the swf9 flex compiler's 'incremental' compilation option.
>
> The flex incremental compiler uses timestamps on the .as files, and a
> cache db file of these timestamps, to determine which files to
> recompile.
>
> So this change makes the script compiler write the intermediate .as
> files to a deterministically named directory
> (tmp/lzswf9/build/path-to-app/) and tries to ensure that only files
> whose content is actually changed are modified on disk.
>
> For speed, this change adds an in-memory cache to the swf9 script
> compiler of every intermediate .as file (which could get large I
> imagine) to decide whether the content has changed on disk.
>
>
> + adds 'compiler.swf9.incremental' flag to lps.properties
>
> - This change doesn't add any way to pass the 'incremental' flag in
> via query arg or command line args to the tag compiler. The only way
> to set it right now is via the lps.properties file.
>
> Tests:
>
> compile a large test app with 300 view-classes (views with
> constraints, that produce anonymous classes). The first compile takes
> 20 seconds, the second incremental compile takes 16 seconds.
>
> About half the time is spent in the tag compiler and script compiler,
> and the other half the time spent in the flex compiler (for the first
> compile).
>
> More specifically when I call just the flex compiler by hand, the
> first compile takes about 9 seconds, and the second (incremental)
> compile takes about 2 seconds.
>
> So something is eating another three seconds, I think it is my code
> which string-compares each cached javascript class file to determine
> whether to rewrite the file on disk.
>
>
>
> Modified: openlaszlo/trunk/WEB-INF/lps/config/lps.properties
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/config/lps.properties 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/config/lps.properties 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -166,8 +166,9 @@
> compiler.swf9.execflex=false
> # Tell swf9 compiler to catch errors in debug mode
> #compiler.swf9.catcherrors=true
> +# Use the incremental compilation feature of swf9 compiler (experimental)
> +compiler.swf9.incremental=false
>
> -
> #===============================================================================
> # i18n locale parameter
> #i18n.locale=ja_JP
>
> Modified: openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/Parser.java
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/Parser.java 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/Parser.java 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -27,7 +27,6 @@
> import org.xml.sax.SAXException;
> import org.xml.sax.XMLFilter;
> import org.xml.sax.helpers.XMLFilterImpl;
> -import org.apache.commons.collections.LRUMap;
> import org.openlaszlo.server.*;
> import org.openlaszlo.utils.*;
> import org.openlaszlo.xml.internal.*;
> @@ -740,11 +739,6 @@
> }
> }
>
> - /** Cache of compiled schema verifiers. Type Map<String,
> - * org.iso_relax.verifier.Schema>, where the key is the string
> - * serialization of the schema. */
> - private static LRUMap mSchemaCache = new LRUMap(1);
> -
> void saveStartLocation (Element elt,
> String pathname,
> String messagePathname,
>
> Modified: openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/SWF9Writer.java
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/SWF9Writer.java 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/compiler/SWF9Writer.java 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -387,6 +387,10 @@
> addScript("canvas.initDone()");
>
> Properties props = (Properties)mProperties.clone();
> +
> + // TODO [hqm 2009-01] make this a compiler command line option, (add to CompilationEnvironment)?
> + props.put(org.openlaszlo.sc.Compiler.INCREMENTAL_COMPILE, LPS.getProperty("compiler.swf9.incremental"));
> +
> // Set up the boilerplate code needed for the main swf9 application class
> props.put(org.openlaszlo.sc.Compiler.SWF9_APPLICATION_PREAMBLE, makeApplicationPreamble());
> props.put(org.openlaszlo.sc.Compiler.SWF9_APP_CLASSNAME, MAIN_APP_CLASSNAME);
> @@ -402,9 +406,13 @@
> ScriptCompilerInfo compilerInfo = new ScriptCompilerInfo();
> props.put(org.openlaszlo.sc.Compiler.COMPILER_INFO, compilerInfo);
>
> + // Working directory path to place intermediate .as3 files
> + compilerInfo.buildDirPathPrefix = mEnv.getLibPrefix();
> + System.err.println("compilerInfo.buildDirPathPrefix = "+mEnv.getLibPrefix());
>
> try {
> scriptWriter.close();
> +
> byte[] objcode = ScriptCompiler.compileToByteArray(scriptBuffer.toString(), props);
> InputStream input = new ByteArrayInputStream(objcode);
>
> @@ -513,6 +521,10 @@
> }
>
> Properties props = (Properties)mProperties.clone();
> +
> + // TODO [hqm 2009-01] make this a compiler command line option
> + props.put(org.openlaszlo.sc.Compiler.INCREMENTAL_COMPILE, LPS.getProperty("compiler.swf9.incremental"));
> +
> // Pass in the table of lzx class defs
> props.setProperty(org.openlaszlo.sc.Compiler.SWF9_APPLICATION_PREAMBLE, makeLibraryPreamble());
> props.put(org.openlaszlo.sc.Compiler.SWF9_APP_CLASSNAME, LIBRARY_CLASSNAME);
>
> Modified: openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/Compiler.java
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/Compiler.java 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/Compiler.java 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -480,6 +480,7 @@
> public static String GENERATE_FUNCTION_2 = "generateFunction2";
> public static String GENERATE_FUNCTION_2_FOR_LZX = "generateFunction2ForLZX";
> public static String GENERATE_PREDICTABLE_TEMPS = "generatePredictableTemps";
> + public static String INCREMENTAL_COMPILE = "incrementalCompile";
> public static String INCLUDES = "processIncludes";
> public static String INSTR_STATS = "instrStats";
> public static String LINK = "link";
>
> Modified: openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9External.java
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9External.java 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9External.java 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -16,8 +16,11 @@
> import java.util.regex.Pattern;
>
> import org.openlaszlo.sc.parser.*;
> +import org.openlaszlo.utils.FileUtils;
> import org.openlaszlo.server.LPS;
>
> +import org.apache.commons.collections.LRUMap;
> +
> /**
> * The SWF9External manages communication with the
> * external compiler - generation of source files,
> @@ -42,6 +45,15 @@
> public static final String WORK_DIR_PARENT = "lzswf9";
>
> /**
> + * Used for incremental compilation option of flex compiler. This
> + * table stores a copy of previously compiled .as file, so we know
> + * whether it has changed before we update the file on disk.
> + *
> + */
> +
> + public static final Map sIncrementalCompilationCache = Collections.synchronizedMap(new LRUMap(10000));
> +
> + /**
> * The prefix for naming the work directories, which appear
> * under the WORK_DIR_PARENT in the Java runtime's temp dir.
> * For example, /tmp/lzswf9/lzgen...., although /tmp
> @@ -69,7 +81,7 @@
>
> private int maxSubdirnum = 0;
>
> - public SWF9External(Compiler.OptionMap options) {
> + public SWF9External(Compiler.OptionMap options, boolean buildSharedLibrary) {
> this.options = options;
> mInfo = (ScriptCompilerInfo) options.get(Compiler.COMPILER_INFO);
> if (mInfo == null) {
> @@ -79,14 +91,31 @@
> // Re-use the previous working directory from the ScriptCompilerInfo
> workdir = mInfo.workDir;
> } else {
> - workdir = createCompilationWorkDir();
> + workdir = createCompilationWorkDir(options, buildSharedLibrary);
> // Copy pointer to working directory to the ScriptCompilerInfo,
> // so any subsequent <import> library compilations can use it.
> mInfo.workDir = workdir;
> }
> - //System.err.println("REUSE_WORK_DIRECTORY = "+options.getBoolean(Compiler.REUSE_WORK_DIRECTORY) + " workdir = "+workdir);
> +
> + // If this is not an incremental compile, erase all files in the working directory
> + if (!options.getBoolean(Compiler.INCREMENTAL_COMPILE)) {
> + deleteDirectoryFiles(workdir);
> + }
> }
>
> + public static void deleteDirectoryFiles(File dir) {
> + if (dir.isDirectory()) {
> + File[] children = dir.listFiles();
> + for (int i=0; i<children.length; i++) {
> + File f = children[i];
> + if (f.isFile()) {
> + f.delete();
> + }
> + }
> + }
> + }
> +
> +
> /**
> * Return the bytes in a file
> */
> @@ -133,7 +162,7 @@
> * and return a File for it.
> * @throw CompilerError when directory creation fails
> */
> - private File createCompilationWorkDir()
> + private File createCompilationWorkDir(Compiler.OptionMap options, boolean buildSharedLibrary)
> {
> // TODO: [2007-11-20 dda] Need some provisions for file
> // cleanup on error, and on success too.
> @@ -143,12 +172,20 @@
> String tmpdirstr = System.getProperty("java.io.tmpdir");
> String swf9tmpdirstr = tmpdirstr + File.separator + WORK_DIR_PARENT;
> (new File(swf9tmpdirstr)).mkdirs();
> -
> - f = File.createTempFile(WORK_DIR_PREFIX, "", new File(swf9tmpdirstr));
> - if (!f.delete())
> - throw new CompilerError("getCompilationWorkDir: temp file does not exist");
> - if (!f.mkdir())
> - throw new CompilerError("getCompilationWorkDir: cannot make workdir");
> + String appDirPrefix = mInfo.buildDirPathPrefix;
> + if (buildSharedLibrary) {
> + // Compiling the LFC
> + f = File.createTempFile(WORK_DIR_PREFIX, "", new File(swf9tmpdirstr));
> + if (!f.delete()) {
> + throw new CompilerError("getCompilationWorkDir: temp file does not exist");
> + }
> + if (!f.mkdir()) {
> + throw new CompilerError("getCompilationWorkDir: cannot make workdir");
> + }
> + } else {
> + f = new File(swf9tmpdirstr + File.separator + appDirPrefix);
> + f.mkdirs();
> + }
> }
> catch (IOException ioe) {
> throw new CompilerError("getCompilationWorkDir: cannot get temp directory: " + ioe);
> @@ -973,6 +1010,9 @@
> cmd.add("-target-player=10.0.0");
> }
>
> + if (options.getBoolean(Compiler.INCREMENTAL_COMPILE)) {
> + cmd.add("-incremental=true");
> + }
>
> if (!buildSharedLibrary) {
> String mainclassname = (String) options.get(Compiler.SWF9_WRAPPER_CLASSNAME);
> @@ -1100,6 +1140,22 @@
>
> FileOutputStream fos = null;
>
> + String content = pre + body + post;
> + File diskfile = new File(infilename);
> + if (options.getBoolean(Compiler.INCREMENTAL_COMPILE)) {
> + if (diskfile.exists()) {
> + String oldfile = (String) sIncrementalCompilationCache.get(infilename);
> + // If old contents of file are the same, don't rewrite the file
> + if (oldfile != null) {
> + if (content.equals(oldfile)) {
> + return;
> + }
> + }
> + }
> + // If we wrote a new file, enter the contents in the cache;
> + sIncrementalCompilationCache.put(infilename, content);
> + }
> +
> try {
> fos = new FileOutputStream(infilename);
> fos.write(pre.getBytes());
>
> Modified: openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9Generator.java
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9Generator.java 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/SWF9Generator.java 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -651,7 +651,7 @@
> {
> boolean hasErrors = false;
> boolean buildSharedLibrary = options.getBoolean(Compiler.BUILD_SHARED_LIBRARY);
> - SWF9External ex = new SWF9External(options);
> + SWF9External ex = new SWF9External(options, buildSharedLibrary);
>
> for (Iterator iter = tunits.iterator(); iter.hasNext(); ) {
> TranslationUnit tunit = (TranslationUnit)iter.next();
> @@ -703,7 +703,7 @@
> }
>
> /**
> - * @copyright Copyright 2006-2008 Laszlo Systems, Inc. All Rights
> + * @copyright Copyright 2006-2009 Laszlo Systems, Inc. All Rights
> * Reserved. Use is subject to license terms.
> */
>
>
> Modified: openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/ScriptCompilerInfo.java
> ===================================================================
> --- openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/ScriptCompilerInfo.java 2009-02-20 05:45:03 UTC (rev 12962)
> +++ openlaszlo/trunk/WEB-INF/lps/server/src/org/openlaszlo/sc/ScriptCompilerInfo.java 2009-02-20 06:42:40 UTC (rev 12963)
> @@ -3,7 +3,7 @@
> * ****************************************************************************/
>
> /* J_LZ_COPYRIGHT_BEGIN *******************************************************
> - * Copyright 2001-2006, 2008 Laszlo Systems, Inc. All Rights Reserved. *
> + * Copyright 2001-2006, 2008, 2009 Laszlo Systems, Inc. All Rights Reserved. *
> * Use is subject to license terms. *
> * J_LZ_COPYRIGHT_END *********************************************************/
>
> @@ -18,5 +18,14 @@
> public File mainAppWorkDir = null;
> /** Path to a (library) compile's as3 working directory */
> public File workDir = null;
> +
> + public String buildDirPathPrefix = null;
> +
> + public String toString() {
> + return "{ScriptCompilerInfo: mainAppWorkDir="+mainAppWorkDir+
> + ", workDir="+workDir+
> + " buildDirPathPrefix="+buildDirPathPrefix+"}";
> + }
> +
> }
>
>
>
> _______________________________________________
> Laszlo-checkins mailing list
> Laszlo-checkins at openlaszlo.org
> http://www.openlaszlo.org/mailman/listinfo/laszlo-checkins
--
Regards,
Max Carlson
OpenLaszlo.org
More information about the Laszlo-checkins
mailing list