Creating Source Code From Abstract Representations

Contents

• What is this project?
• Inspiration
• Try it for yourself
• How It Works
  • Overview
  • Abstract Syntax Trees
  • Custom Parser
  • Database Integration
  • HTML
• Testing
• Contributing
• Closing Thoughts

What is this project?

This project is a tool that is used to produce human readable and functional code for many different app frameworks from one higher level abstracted version of the app. The project is still very experimental at this stage and only works with limited functionality. The project is open source and can be downloaded and extended by any who wish to. The project can be found on GitHub and the NPM package can be found here.

Inspiration

Creating apps for multiple platforms and frameworks can be hard. There are so many different frameworks, on the web alone you have basic JavaScript, React, Ruby on Rails and many more. That doesn't even go into the huge market that is mobile apps. The motivation behind this project is to make the process of creating apps for many different platforms and frameworks as easy and as simple as possible.

The biggest aim is to have the ability to 'write once, produce many'. This means that amazing ideas that people have shouldn't be suppressed by the daunting task of learning many different languages/frameworks. This can be especially true for less experienced people who want to produce something but lack the technical skills to do so. That is where this project comes in. You learn one framework and language and can produce functional code for many different frameworks.

The second aim of this project is to have an easy to use framework so even those with less technical skills can leverage its capabilities. This includes abstracting a lot of the basic features needed for a CRUD (Create, Read , Update, Delete) application into english readable function names. This also means that the method for creating a UI should be simple and easy to integrate with the tool.

Try it for yourself

This is a CLI tool, written in JavaScript running on Node.js. The way it works is that you create your app using the abstract-app framework. You have a master app which you will write and then more framework specific versions will be produced from this. The tool implements a MongoDB database on the backend, it can generate all the code needed for this to work. For the time being the app must be written in python. A misconception many have is that you run the python. That is incorrect, it is used for its easy to understand syntax and will be converted into JavaScript. HTML can be provided from either creating a site using Webflow and pulled into the project given the URL of the site, or local HTML can be used now is passed in to the tool in the same way Webflow HTML is. The tool contain an example messaging app styled after Discord. It can be compiled right away.

Prerequisites

To ensure it works correctly you will need to have Python and Node.JS installed.

Once python has been installed, open a terminal and run pip install javascripthon. This is a library needed to convert the Python into JavaScript.

At the time of writing this tool has been tested with the most recent versions of Python and Node.JS which are 3.11.2 and 18.13.0 respectively. It has only been tested on Windows 11 to date.

If you do not know python a good tutorial can be found here. You don't need to know JavaScript to use this tool but understanding some basics may help when looking at the generated code. A good JavaScript tutorial can be found here.

Downloading

The tool can be downloaded in two ways, the first is downloading the source directly from the GitHub repository. The second is to download it through npm (requires Node.JS). This blog post will cover how to use the npm method.

1. Go to the package. It specifies the command to run as npm i abstract-app-to-source, however I would highly recommended running npm i abstract-app-to-source -g instead, this will install the package globally and allow you to run it from anywhere.
2. After running this command you should see the following:

   

3. In your terminal run npx abstract-app --help this verifies the package is downloaded correctly:

   

Creating An App

I will show you how to create a very simple messaging app styled in a similar way to Discord, all created with the abstract-app framework. All the code for this example is available in the GitHub repo. To get started we will generate our HTML, I already have a Webflow project set up that you can view in read only mode, to see how it is structured. If you want to try Webflow with this tool sign up for a free account. This post will not show you how to use webflow, so a good tutorial can be found here going over the basics, I will focus more on how to use webflow with this specific tool.

Upon opening the project you will see the webflow editor, to leave the preview view you are in by default press the button to leave this view. We will now be in editor view, this allows us to examine the project and make changes, as this is read only any changes made will not be saved.

Open the navigation panel as seen below:

Upon examining this project you can see the main parts we have:

• A Text Block to display our messages to
• A form to enter a name and a message to send

The tool works similar to JavaScript as it uses the IDs of elements to assign events, get data and set output. If you click on the Text Block you can see it has a unique ID. Anything you want to interact with must be given an ID.

For our input boxes we must go further. A limitation of webflow is that it does not allow inputs to be outside of form blocks. This causes issues as if we want an input but do not want it to POST data to the server every time we enter data to it. To fix this issue the tool can modify HTML at compile time. To do this we must add a custom attribute to a Form Block element. This will tell the tool that we want to treat this form as a series of inputs and buttons rather than a form, this allows these forms to be used without posting data back to the server every time we enter data.

Click on the Form Block element and add the custom attribute fake-form="true".

Publishing HTML

If you are using your own project you must publish the webflow site so it is accessible to the tool. To do so click on the publish button in the top right of the editor and click publish. If you are just following along with this tutorial just use https://example-site-58b8c9.webflow.io/ which is the same as the example. Alternatively the local HTML can be found here. If you are using your own site you must publish each time after a change is made for the tool to detect the changes.

Code

The aim of our app is to be able to send messages between different clients, there is many different ways to do this, but we will be utilizing the MongoDB integration this tool includes. The code for this tutorial can be found here. The documentation for the framework explains how each method works and the correct syntax to use.

To make a very simple messaging app we rely on two things, sending messages and receiving messages. To send messages we will take the users name and message data and add it to the database. To receive messages we can subscribe to the insertChange event. This event fires when a new record is inserted into the database. Using this framework you do not need to write any back end database code yourself as it will all be auto generated.

                                    def start_up():
    __COMMENT__("Set error text to none")
    setOutput_errorTxt("")

    __COMMENT__("Load all the messages in the database")

    allMessages = dbLoadData("Discord", "messages", {})
    output = ""
    
    if allMessages == "CONN_ERR":
        output = NOT_CONNECTED_MSG
    else:
        __COMMENT__("Loop through all the messages and add them to the messages array")
        
        for message in allMessages:
            list(messages).append(message)

        __COMMENT__("Get all the messages")
        output = createMessagesString()

    setOutput_msgTxt(output);

                                

The above code is a snippet from the start_up method, a custom method that runs when the app starts (or loads the webpage). The first thing we do is load all the messages from a database called Discord and a collection called messages. This dbLoadData method returns a list of all the messages. Each message is an object containing the sender and the message data.

Next we check if the messages does not equal "CONN_ERR", this indicates that our app is not connected to the database. We can then set the output to an appropriate error message.

If we are connected to the database then we can create a string containing each message on a new line (the tool does not support appending to text at the time of writing, a quick hack is to create a string with new lines).

Finally using the setOutput_ID method we can set the output of the msgTxt element to the correct output.

                                    def sendMessage_click_sendMsgBtn():
    __COMMENT__("Get the name and message data\nCheck if they are not empty")

    nameValue = getValue_nameInput()
    msgContent = getValue_msgInput()

    output = ""

    if (checkWhitespace(nameValue) or checkWhitespace(msgContent)):
        output = "Please enter a name and message"
        setOutput_errorTxt(output)
    else:
        __COMMENT__("Send to the database")
        dbInsertData("Discord", "messages", { "sender" : nameValue, "msg" : msgContent })
        setOutput_errorTxt(output)

                                

To send messages we use the above code. We first need to get the name and message the user entered, this can be done using the getValue_ID method. We then check if the name or message is just whitespace, if so we can display an error message to the user. If the name and message are valid we use the dbInsertData method to add the new message to the database.

                                    def insertChange():
    __COMMENT__("Upon insert update the messages")
    print("Detected an output")
    messages = dbLoadData("Discord", "messages", {})
    output = createMessagesString()
    setOutput_msgTxt(output)

                                

For the final piece of this program we need to read any new messages. We can listen for the insertChange method and then read the new data from the database when it is fired. Similarly to the start_up method we create a string with new lines separating the messages.

Creating a Database Server

Creating a database server is very simple, we can just run the command npx abstract-app gen-db --location "Location to create sever". This command will generate a server that will automatically start a MongoDB instance. A database folder will be created, you do not need to create a database, when you write to it for the first time one will be made automatically by MongoDB. The location must be a directory that does not already exist. If it does then you will be asked if it is ok to delete the directory that exists there.

Above is what the directory should look like when the server is created correctly. To run the server simply double click the RUN_SERVER.bat file. At the time of writing the server can only be run on Windows.

Creating a Web App

To create a web app all you need is one command, to view the help for this command run
npx abstract-app gen-web-app --help.

To generate the app simply run
npx abstract-app gen-web-app --name "Location or name of project" --html_address "Address of HTML, webflow or local" --app_path "path to python app"

Above is what the structure of a generated web app should look like. To run the web server just enter the directory and run the command node ./app.js. Open a web browser on the same computer and go to localhost:80. At the time of writing all apps try and connect to a database server on localhost:3000. Below is what the web app should look like.

Generating an Electron App

To generate an electron app the command is very similar to a web app. The only difference is that where it says gen-web-app, should be changed to gen-electron-app. An example of that is:
npx abstract-app gen-electron-app --name "Location or name of project" --html_address "Address of HTML, webflow or local" --app_path "path to python app"

Above is how the electron directory should look. To start the electron app simply enter this directory and run npm start. Below is what the app should look like.

Working App

Below is a short demonstration of an Electron app and a web app running at the same time. They are both connected to the same database server and are able to send and receive messages.

How It Works

If you are interested in learning how this works and possibly contributing to the development then please keep reading.

Overview

In general terms the way the project works is by taking the python code you write and transpiles it into JavaScript. This JavaScript is then parsed by my custom parser. This looks for methods and functions defined using the abstract-app framework outlined in the docs. Once it has found any of these methods or function definitions it will produce source code for the specific framework you want. At the time of writing it is capable of producing apps for the Electron Framework and a web app built on top of Express.js. Both of these apps contain MongoDB integration, the framework provides a number of methods to read, write, update and delete from a database. It is also capable of listening for when events occur within the database. The beauty of the tool is that the parser handles all this database code for you.

Abstract Syntax Trees

Abstract Syntax Trees is way of representing source code in a manner that is free from the strict syntax of programming languages but maintains the structure of the program.

ASTs (Abstract Syntax Trees) are a vital part of this project, they allow for the traversal of the source code to find important parts and can then be compiled back into source code.

As there is no universal AST format the first problem was converting from a Python AST to a JavaScript one. There is a few ways that we could do this, these include:

• Write a Python AST to JavaScript AST converter or find a library to do it for us
• Use an existing transpiler to convert Python code into JavaScript then into a JavaScript AST

So the second option is what I went with. As I have mentioned above I am using the JavaScripthon library to transpile our Python code into JavaScript code. Then we use an existing parser to convert the JavaScript code into an AST that can then be parsed by my custom parser.

If you want to learn more about ASTs and how to write a traversal algorithm then look at this guide which I have written in the past.

Custom Parser

The bulk of this application is to parse the AST produced from the Python code. If look at the source code the ParsePythonTree.js file you will see a method called parsePythonTree(tree) . This function is used to parse the AST and produce the appropriate changes.

There is two main ways that the AST is parsed. One is looking for function definitions and the other is looking for function calls. We will first look at how it parses function calls. Again looking at ParsePythonTree.js locate the method called manipulateSetOutput(tree);. This function is called from within parsePythonTree(tree); . The definition of manipulateSetOutput(tree); is seen below.

                                function manipulateSetOutput(tree){
    console.log("GETTING OUTPUT");
    visit(tree, (node, parent, key) =>{
        let check = checkSetOutput(node);
        if (check !== null){
            console.log("IS AN OUTPUT");
            let code = constructSetOutput(check);
            console.log("CODE IS:");
            console.log(code);
            node.type = code.type;
            node.expression = code.expression;
        }
    });
}                        

                            

Looking at this method you will see a function called visit. This method is explained in my guide as mentioned above. What this function does is perform a depth first search on the the AST and checks if any node follows the format of a setOutput function. It will then extract the data needed and construct a custom AST based on the data provided.

The checkSetOutput(node) function looks as below:

                                // checks if a method call matches the form of setOutput_id
// and then return an object that contains the data
function checkSetOutput(node){
    let outPutData = null;
    let check = /^setOutput_[^_]*$/;
    // check if the node is an expression
    if (node.type === "ExpressionStatement"){
        if (node.expression.type === "CallExpression"){
            if (node.expression.callee.type === "Identifier"){
                if (check.test(node.expression.callee.name)){
                    // test the name of the method against the above regex
                    let callData = node.expression.callee.name.split("_");
                    console.log(callData[1]);
                    // splits based on the _ and the second element is the id
    
                    // check if the args contain only one thing
                    let args = node.expression.arguments;
                    console.log(args[0]);
                    if (args.length === 0)
                        return outPutData;
    
                    // now create the node
                    outPutData = 
                    {
                        id : callData[1],
                        data : args[0]
                    }
                }
            }
        }
    }
    return outPutData;
}                  

                            

Within this function we contain nested if statements to check if the AST nodes match that of a function call. We are using some very basic regex to check the format of the function names. If the node we are visiting is confirmed to be a setOutput(data) method we then split the method name on the _. As I mentioned earlier the _ within the function names are very important this is why. Next we need to check the number of arguments that is provided in the function call. Our function only contains one argument, data. If this function has more than one argument, then it is not valid and will be rejected. If after all those checks pass we create an object with the ID and the data.

If we look back at manipulateSetOutput(data), we can see that if our checks pass we start constructing a setOutput node. This method creates a custom AST that is equivalent to the JavaScript below:

                                document.getElementById('id').innerText = data;

                            

Once we have constructed our event it is slotted into the AST at the point out setOutput method is called.

                                node.type = code.type;
node.expression = code.expression;

                            

You would think that we would just set the current node to the new node object. This does not work in JavaScript, we must set the type and expression of the node to the values of the new node.

We have just covered one of the functions here that is used to locate and manipulate a part of the framework. If I was to explain all the manipulate functions this article would be the size of a novel. Take a look yourself and try to understand them. They work in a very similar way to this function, in that it checks if the method names are correct and then using the data in the name constructs an AST node from it. One thing to note is that these other manipulate functions create the nodes from a string JavaScript using the acorn parser as they are simpler. I cover using the acorn parser in my AST guide.

Events

They way events work within the framework is similar to many other frameworks. Events must be declared using a function definition. We saw an example of this above when we created an event for our convert button.

An example of an event listener in JavaScript looks as below:

                                document.getElementById('id').addEventListener('click', functionRunOnFire);

                            

Similar to the setOutput and getValue functions we abstract out the data which we need. This is just the id of the element to attach the event to, the event to listen for such as click in this example and the function to run when the event fires.

That looks something like below

                                def nameOfFunc_event_id():
    # code that runs when the event is fired

                            

Again these function definitions are handled within the parsePythonTree(tree) method within the ParsePythonTree.js file.

                                if (checkEvent(tree.body[i])){
    console.log("in event");
    let eventCode = constructEvent(getEvent(tree.body[i]));
    code.events.push(eventCode);
    code.nonKeyCode.push(tree.body[i]);
    console.log(code.events);
}

                            

The code above shows hwo to parse an event declaration. This does not use the visit function from above as each function declaration will be a top level node meaning there is no need to visit every child node. As can be seen it uses a similar structure to the above methods. We have a function to check the structure of a node and one to construct an event node for us. One main difference is that we are adding our event node to an array within an object called code. This is so that we can extract the code created when parsing.

                                else{
    code.nonKeyCode.push(tree.body[i]);
}

                            

The above code is very important. When iterating through every top level node we add all code that is not identified as an event and doesn't need further manipulation. Everything will be added to this array and then returned from the function. After this 'code' object is returned all the newly manipulated nodes will be added together into a clean AST that is then converted into source code.

Database Integration

As I have said earlier the framework includes integration with MongoDB. At the time of writing this is limited to a local database, and the only functions supported are read, write, update and delete. It also supports listening for new inserts, updates or deletes in the database. As it is built upon MongoDB, a NoSQL database, there is no need to know any SQL for data operations.

Creating the database methods works in a very similar way to the normal HTML manipulation methods. The python code is transpiled into a JavaScript AST, it is then parsed using another custom parser to look for certain method names, these can then be manipulated as needed.

MongoDB Server

Before any of that is done we first need a central server that our apps can talk to. MongoDB already includes a server for running a local database, however trying to set this up and get working can be a challenge in itself. That is why my tool will automatically create a custom server for MongoDB that can be generated.

To check out the sever you can just run the command npx abstract-app gen-db --location "My Server". This will generate the necessary files for a server to run. To run the server simply click the RUN_SERVER.bat file. This will start the MongoDB database and will start a Socket.IO server. If you are not familiar with Socket.IO, it is just an easy to use sockets library that works well with many different languages. The socket server can now listen for incoming requests from apps and carry out the correct database operation. The server is also capable of sending out events to the apps when a change is detected in the database.

To generate the server code I simply have a copy of it in the project that can be seen here, this is copied into the directory of choice when the command is run. The code to generate the rest of the server is very simple and can be seen here. The basic sequence of events is as follows:

1. Check if the directory the user entered does not already exist
   • If it does then ask the user if they want to delete the folder and its contents found there.
2. Create the new folder and the relevant sub folders
3. Copy the MongoDB.exe from the project repository into /bin within the new server folder
4. Copy the server.js file from the templates into the server folder
5. Create the BAT script to run the server
6. Install the necessary packages, these include mongodb and socket.io

Database Methods

To make the access to the database as simple as possible there are a number of methods that can be used to carry out various forms of data manipulation within the database. These include methods for inserting, updating, deleting and loading data. A detailed account of how to use these methods can be found here.

Parsing Database Methods

The parsing of database methods works in very similar way to the normal HTML methods as discussed earlier. The major difference is that the web and Electron apps generate different code.

Electron Database Methods

We will first look at the way electron methods are parsed.

The method we will be focusing on can be found here.

                                function parseElectronDBTree(frontendJS, backendJS, TEST = false)

                            

Above is the definition for the Electron database parsing function. As can be seen we pass in reference to the front and back end AST object. This allows us to use the visit() to traverse the front end tree and locate the appropriate method names. The TEST parameter is used to disable comments output for easier comparisons.

                                    // handle load data nodes
    visit(frontendJS, (node, parent, key) => {
        let loadDataNode = checkLoadDataNode(node);
        if (loadDataNode !== null){
            console.log("Found a load data node");
            // create ipc code needed
            addipcLoadData(backendJS, loadDataNode, node);
        }
    });

                            

The above code is for locating and manipulating a dbLoadData method. If this looks familiar, that's because it is. These database methods follow the same principle as the normal HTML methods, of locate the correct nodes, then manipulate it correctly.

                                function checkLoadDataNode(node){
    // a load data node : DBLoadData("collection name", dict);
    //console.log("Looking for a load data node");

    if (node.type === "CallExpression"){
        if (node.callee.type === "Identifier"){
            if (node.callee.name === "dbLoadData"){
                console.log("Found a load data node");
                // get the args, should be 2
                let args = node.arguments;
                if (args.length !== 3 || args[0].type !== "Literal" || typeof args[0].value !== "string" ||
                    args[1].type !== "Literal" || typeof args[1].value !== "string" || args[2].type !== "ObjectExpression"){
                    throw new Error(`dbLoadData must follow pattern dbLoadData("DB Name", "collection name", dict)`);
                }
                // split the data as needed
                return { 
                    dbName : args[0].value,
                    collectionName : args[1].value, 
                    searchDict : ASTToCode(args[2]) 
                };
            }
        }
    }
    return null;
}

                            

Above is the relevant code for the checkLoadDataNode method. All this method does is check that a function call exists, with the correct name. As the dbLoadData method has 3 arguments that need to be provided, these include the database name, the collection name and the filter.

                                if (args.length !== 3 || args[0].type !== "Literal" || typeof args[0].value !== "string" ||
    args[1].type !== "Literal" || typeof args[1].value !== "string" || args[2].type !== "ObjectExpression"){
    throw new Error(`dbLoadData must follow pattern dbLoadData("DB Name", "collection name", dict)`);
}

                            

Using this somewhat large if statement we can check if the arguments provided match the correct number and data types. In this we check for the inverse of the correct conditions, we can then throw an appropriate exception to the user when we detect a syntax error.

                                // split the data as needed
    return { 
        dbName : args[0].value,
        collectionName : args[1].value, 
        searchDict : ASTToCode(args[2]) 
    };

                            

If our checks succeed and we have a syntactically correct method we can then return the data about it. In this case we return an object containing the different arguments. The searchDict is converted to a code string as we no longer need its AST as we will be inserting it into strings later.

Next is to generate the code needed for inter process communication between the front and back ends of Electron. If you are not familiar with IPC in electron check this out. The function we will be looking at is the addipcLoadData method. This method takes three parameters, these are the front and back end ASTs and the current node data.

                                // backend ipc event listener
    let backendIPC = `ipcMain.on('${eventID}', async function(event, dbName, collName, searchData){
    let res = await readData(new BufferData(dbName, collName, searchData, DataType.read));
    if (res === false)
        event.returnValue = "CONN_ERR";
    else
        event.returnValue = res;
});`;

                            

Above is an example of the code that is to be generated for the backend of an Electron database method. All this simply does is create an event listener using the Electron IPC framework, this listens for the relevant events coming from the front end code. When the event fires, we call the custom method readData, this method simply calls the external server we created earlier using socket.io and sends the appropriate request to load data. We can then return the relevant response based on if this method succeeds of fails.

                                let frontEndCode = `ipc.sendSync('${eventID}', '${loadDataObj.dbName}', '${loadDataObj.collectionName}', ${loadDataObj.searchDict})`;

                            

After creating the back end code we create the front end code as required. Above shows what code is created for the front end. It is much simpler than the back end, all it does is send data to the back end through the Electron IPC framework.

After both of these code strings have been created we can convert them to ASTs and insert them back into there respective trees. This again is very similar to how the normal HTML methods are parsed.

I have only covered the dbLoadData method in this post, but the other database manipulation methods work very similarly.

Web Database Methods

Generating the correct code for a web app takes a different approach to Electron. As our web app is running on an Express.JS server there is no such thing as IPC, instead I am using HTTP to send requests to the Express server, the server then sends requests to the database server through socket.io to manipulate the data as needed.

The appropriate method for parsing the web database code can be found here. An obstacle that had to be overcome was the ability to give the impression of synchronous programming on an asynchronous environment. When creating an Electron App we can write our IPC code synchronously so that the application will wait for responses. This is possible when sending HTTP requests to a server but is very strongly NOT recommended. In order for the tool to work and to be kept simple the user should not be exposed to asynchronous code as it can be difficult to understand, especially for newcomers. One solution was to use the callback methods provided with the JQuery POST method, the tool could then go through and place any subsequent code in the callback. This has some major issues, the response of the POST request could easily be lost if the user didn't use the correct names and it is very difficult to place the code after a request into the callback, especially if it is in a loop. The second option which I have gone with is to make every function asynchronous and then place an await in front of them. This solution works much better as it is just a matter of going through all the method names and making every one async. I can then compare this list of method names to any method call and add the correct await to the call.

                                visit(frontendJS.body, (node, parent, key) => {
    if (node.type === "FunctionDeclaration" && node.id.name !== "BufferData"){
        // if function make async
        console.log("Found func def");
        node.async = true;
        functionDefNames.push(node.id.name);
    }
});

                            

The above code is responsible for finding all method declarations and making them asynchronous.

                                // iterate again to find any functions and make them await
visit(frontendJS.body, (node, parent, key) => {
    if (checkCall(node, parent)){
        if (functionDefNames.includes(node.callee.name)){
            manipulateAwaits(node);
        }
    }
});

                            

The above code is responsible for adding await to the front of each async method. The manipulateAwaits method simply adds the await in the correct position in a node.

Now that we have changed the methods correctly we can get to the manipulation. The web parser tries to minimize the amount of code generated as this reduces the errors. The web parser simply checks that the methods are syntactically correct. The code for the POST is already predefined in a template file. This JavaScript is added to the AST of the front end and provides helper methods that is called to carry out the appropriate requests.

                                // iterate again to check the format of the db methods
visit(frontendJS.body, (node, parent, key) => {
    checkLoadDataNode(node);
    checkInsertDataNode(node);
    checkUpdateNode(node);
    checkDeleteNode(node);
});

                            

Above is the code to validate the syntax of any database methods that the parser finds. These functions are the same as are used in the Electron parser. They will just check for correct syntax and throw an error if any syntax is incorrect.

In hindsight this method is superior to the method used to generate the Electron code, it reduces the need to generate so much redundant code and have it already predefined. The priority was to get a working version so changing it at this stage is not important, but the first thing to be changed when making and additions should be this.

Database Events

A useful feature of many NoSQL databases is the ability to listen for changes that occur within the database. MongoDB supports such operations but is is limited on a local database and hard to set up. I decided the best way forward was to write a simple events system. The way it works, is the database server waits for incoming requests, when it receives a request it will emit an event over socket.io. This event is received by both the Electron and web apps, the apps can then respond appropriately to the event.

                                // makes the changes for an updateChange method
function manipulateUpdateChange(frontendJS, backendJS){
    let backEndCode = `socket.on("update_change", (data) => {
        console.log("Update change");
        win.webContents.send("update_change");
      });`;

    let frontEndCode = `ipc.on("update_change", (event) => {
        updateChange()
      });`;

    // add to the end of backendJS
    addNodeToEnd(backendJS, codeToAST(backEndCode)[0]);

    // add to the end of 
    addNodeToEnd(frontendJS, codeToAST(frontEndCode)[0]);
}

                            

When generating a listener for the change events we follow the same pattern as other methods, in that we look for the method name, then manipulate the code appropriately. Above is code showing how the listeners for an updateChange() method in electron is generated. For the back end it is very simple, it waits for an "update_change" event to be emitted from the database server. It then contains code to send data back to the front end. The front end code here is also very simple, it contains an IPC listener that runs the updateChange() method, which is defined by the user. Similar to above we convert our code strings into ASTs and add them to appropriate tree.

For the web app implementation, it uses a slightly different approach. As to maintain some level of security the front end of an application should not interact directly with the database server. But as there is no "back end" the same as an Electron app, just a server, we cannot send the events straight to the front end. That is why the Express.JS web server will also be running another socket.io server. The Express server will then receive the change events and send them on to the front end through another socket.io connection.

                                function manipulateUpdateChangeWeb(frontEndJS){
    const code = `localSocketConn.on("update_change", (data) => {
        updateChange();
    });`

    const AST = codeToAST(code);

    addNodeToEnd(frontEndJS, AST[0]);
}

                            

This method is very simplistic compared to the Electron version. All we are doing is setting up a listener to the local socket.io server. We then turn the code string into an AST and add it to the relevant AST.

For both Electron and web the other changes methods (insert and delete) are generated in the same way as above.

That concludes the database methods, all the code needed can be found in the ParseDatabase.js file.

HTML

The final part of the parser is the HTML. We have seen that HTML can be used from a published webflow site, or a local HTML file. The HTML is less complex than the app code, however we still have to do some manipulation in order for it to work.

Identifying Elements

To be able to access HTML elements from within our app we must give them a unique ID. This is done using the id attribute. This can be done by adding IDs to local HTML elements or through the webflow interface.

                                <button id="myBtn">My cool button</button>
                            

If we wanted to add an event listener to this we would have to use the same ID in app to allow it to work.

                                def myEvent_click_myBtn():
    #code to run on event fire

                            

Inputs

When we were creating our webflow site I said that in order to use an input on its own we had to add a form block and give it a custom attribute called fake-form="true". This is because webflow doesn't support inputs outside of forms. This is a workaround for now, as webflow adds a class that makes forms behave in a certain way. To get around this I am using node-html-parser to parse the HTML and remove the webflow classes that cause issue on any form block that contains the attribute fake-form="true".

The code for this can be found in the HTMLParser.js file. To locate the div elements with the fake-form="true" , it is very simple. We just parse the HTML into an AST then locate all div elements. If the div element has the attribute fake-form="true" then we manipulate it to remove the webflow class. After the whole tree has been traversed the we simply compile it back into a string of HTML.

External Library Imports

Some of the apps require third party libraries to work as expected. These include things like jQuery and socket.io. When using Webflow JQuery is already included in the HTML, that means all we need to do is import socket.io. However when using a local HTML file we need to import both JQuery and socket.io. Similarly when creating an electron app we do not need to include any of these libraries as we do not use any of them on the front end. Below shows a simple implementation of how to import JQuery and socket.io.

                                function addWebIndex(html, isLocalHTML){
    // parse the HTML
    let root = parse(html);

    // add the socket.io import
    root.insertAdjacentHTML("beforeend", '<script src="https://cdn.socket.io/4.6.0/socket.io.min.js"' + 
    'integrity="sha384-c79GN5VsunZvi+Q/WObgk2in0CbZsHnjEqvFxC5DxHn9lTfNce2WW6h2pH6u/kF+"' +
    'crossorigin="anonymous"></script>');

    // check if this is a local HTML file,
    // if it is, we need to include the Jquery import
    if (isLocalHTML){
        root.insertAdjacentHTML("beforeend", '<script src="https://code.jquery.com/jquery-3.5.1.min.js"' +
        'integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0="' +
        'crossorigin="anonymous"></script%gt;');
    }

    // insert at the end
    root.insertAdjacentHTML("beforeend", "<script src='./static/index.js'></script>");

    return root.toString();
}

                            

In the above code we are simply including the socket.io import in the HTML, using the same node-html-parser library as before. This occurs for all web app generations. However when generating a local HTML file we need to include the JQuery import as well.

Obtaining the HTML

Webflow does offer a code export feature but it can be quite expensive. This tool can be used without paying for webflow luckily. As mentioned earlier to see any changes made to a webflow site you must publish it, and supply the URL of the published site as an argument. This is because the tool uses the JavaScript fetch API to get the HTML from the published site. Once we have the HTML we can manipulate it as needed. It is also possible to use local HTML file as well. These can be used for testing purposes or if you want to use this tool for a site that you have not published. To use a local HTML file simply pass the path of the file the same way you would use a webflow link when running the tool.

How to learn more

In this post I have covered what I feel is necessary for someone to understand how this project works. I have not covered every line of code but feel free to dig deeper into the source available on my GitHub. To keep up to date with the newest additions refer here.

Testing

To test the tool I have created a simple test suite. The test program can be found here. The objective of these automated tests is to ensure that the transpilation works as expected. Each test has an input and expected output. To run the tests, clone the GitHub repository and run npm test. You should see a large amount of output, this is not the test output. The test results can be found in a 'Test_Out.log'. This file details the passing and failing tests.

These automated tests aim to check the transpilation is working as expected. They do not check the functionality of apps made using the framework. As there is practically an infinite number of apps that can be made using the framework, it is not possible to test all of them. I have instead decided to test each custom method in the framework individually with various valid and invalid input. This ensures the parser and transpiler are working as expected.

Some parts of the tool cannot easily be tested using automated tests. These include the downloading of Webflow HTMl and the functioning of the database methods. That is why I have a series of manual tests. These run to ensure that the tool is working as expected. To test the database methods I created a very small program that utilizes all of the database methods. The code used for this generation can be found here.

A more detailed explanation of the testing can be found in the test plan document. In it I detail how to add a new automated test and the process for running manual tests.

Contributing

If your are interested in contributing to the development of this project then keep reading below to find out more. When adding a feature be sure to keep in mine the core principle of this project, 'write once, produce many'. This means the ultimate goal for this project is to be able to produce an app in multiple frameworks, having only written it once. Try to add features which you think will be useful to a user wanting to use the tool.

Current Limitations

Below is a list of current limitations that exist within the tool. Some of these can be fixed in the future, however some may not be possible to change.

• Python code must be written in line with what is possible with JavaScripthon. This limitation is acceptable I believe as it is the most reliable way to transpile Python to JavaScript at the minute.
• Can only produce a web app and an Electron app. This is limiting as it means users are missing out on the huge mobile market.
• The tool currently only converts to JavaScript. This is because Python and JavaScript are similar allowing for easy conversion between the two. This issue would be quite hard to fix as it would require a transpiler to be written to go from Python to many languages.
• Database integration is limited to MongoDB. This is because it is a quick and easy database to setup locally, that does not require the use of SQL. This limitation is unlikely to be addressed as it is not a major issue for the project as most of the database code is hidden from the user anyway.
• Advanced queries cannot be carried out on the database, this is because I thought it better to get a simpler version of all the forms of database manipulation working.
• Database deletes and updates are limited to deleting and updating a single record at a time, again this is because I wanted to keep the database methods simple.
• By default any apps will only look for a database server on the localhost.
• The tool is limited to being run through the command line.

That is a list of the most obvious limitations that I have identified for now. I am sure as development proceeds this list will become smaller.

Road Map

Below is a list of features that I would like to add to the tool in the future. This list is not exhaustive, and feel free to add any feature you feel will improve the tool.

• Add support for Capacitor so that mobile apps can be produced.
• Support more HTML manipulation methods, such as appending to text and modifying HTML itself. This will allow more responsive apps to be made.
• Add support for other source languages such as JavaScript, C#, Java, C++, etc. This will allow users who are not familiar with Python to use the tool.
• Add external resources such as the ability to load in images, this cannot be done currently.
• Move away from using the command line to run the tool and instead to a GUI, possibly even a web app, to avoid users having to download any additional software.
• Some sort of configuration file to allow users to change the default settings of the tool, such as the IP address the apps point to for the database server.

Notes for Development

When adding a feature use the following as guidelines:

• Use the same library for parsing code into ASTs and for compiling ASTs back to source code. These being acorn and estree-util-to-js.
• The format of AST used is estree. This will be used for the forseeable future as it is easy to use and good documentation exists for it.
• JavaScript should be written in line with the ES6 format.
• When generating JavaScript code to manipulate the DOM please use the built in JavaScript methods, rather than an external library such as JQuery. This is because some frameworks such as Electron and Capacitor do not support these yet.
• Use the yargs library when adding or making changes to commands.
• When adding any options to the custom parser, please add them to the parsePythonTree(tree) method as shown above.
• When adding a new database method, please add it to the ParseDatabase.js file.
• When adding new database methods try and use the approach outlined for a web app, this consists of creating helper methods and including them in the overall JavaScript rather than trying to generate the code dynamically.
• When developing for the web app use the Express.JS framework.
• When adding a new feature try and add the appropriate tests, for automated tests follow the guidance set out in the test plan

These are merely guidelines to follow when developing, however you are free to fork the project and approach the problem in any way you see fit. A full list of libraries used in development can be found here.

Thank you to anyone who wishes to contribute, as it will help to make this project bigger and better.

Closing Thoughts

I hope this post has been shown you the power of this framework and how it can be used to enhance your own work. It is still in its infancy at this stage, but with continued development it will become something that many people will find invaluable and hopefully be used to make amazing apps.