Convert hand-drawn design to HTML

August 11, 2020 Sunil Ghimire

In this tutorial, we’ll be building a program that is used to convert the hand-drawn design into HTML, by using an algorithm called sketch2code. The converted HTML will be adjusted based on the device-width, and responsive across devices. So, to boost the conversion, the AI algorithm allows the use of the imported designs for continuous learning.

Process Flow

Transformation method of a handwritten image to code, Implementation of this approach is shown below:

1. The user first uploads an image via the website.

2. A model named custom vision determines what HTML components appear in the image and where they are positioned.

3. In the predicted elements, a handwritten text generation service reads the text inside.

4. A model algorithm uses spatial knowledge from all of the projected product bounding boxes to create a grid structure that matches all.

5. An engine of generation HTML uses all these pieces of knowledge to create an HTML markup code that represents the outcome of an image.

The architecture involved to convert the hand-drawn design to HTML

1. A Microsoft Custom Vision Model : This model was trained with images of various handwriting styles tagging details on the most popular HTML elements such as Images, TextBox, and Buttons.

2. A Microsoft computer vision service : A service based on computer vision is used for defining the text written into a design feature.

3. Azure Blob storage : All steps involved in the process of creating HTML are processed, including original image, effects of predictions, and material grouping of the model.

4. An Azure function: It serves as the backend entry point that manages the cycle of generation by communicating with all resources.

5. An Azure web app: User front-end to encourage you to upload a new template and see the effects of the created Code. Those elements form the following architecture: