Microsoft AI Labs has some nice projects to try out on their website. One is the Sketch2Code project. "Sketch2Code uses AI to convert hand-written drawings to working HTML prototypes." Let´s try it out!
The Microsoft AI Lab projects website shows a bunch of interesting projects. Some of them are probably known, others are worth to explore.
The Sketch2Code website informs about the project: Transform any hands-drawn design into a HTML code with AI. Kindly, there´s also a lab that demonstrates how to use Azure Custom Vision to create an application that is capable of transforming a hand-drawn user interface into valid HTML markup available at Sketch2Code Lab!
You can try it out right away at https://sketch2code.azurewebsites.net. There are samples, and you can upload your own drawings as well.
I have drawn a simple contact form, photographed it and uploaded it. Here´s the result (click to enlarge).
Not bad for my handwriting, a bad ballpoint pen, and a raw photo made by phone. The generated code result.html is using Bootstrap 4 for formatting and can be downloaded for further processing.
So, I was curious how the form generator handles other drawings. I drew another sketch note (with a better ballpoint pen), this time a workflow diagram. Here´s the result (click to enlarge).
The boxes are converted to buttons, such as <button class="btn btn-primary">Configure firewall for Microsoft 365</button>. Every box is generated as a button, OCR generates the text.
The predicted object details give some insights of the transformation process.
So, the statement "Sketch2Code is transforming a hand-drawn user interface into valid HTML markup" is is correct. Technically, Sketch2Code is using the Azure Cognitive Services. The Custom Vision and Computer Vision APIs are used for object detection, OCR is used for handwriting recognition. The objects returned help to identify the layout that is passed to the Sketch2Code MVC web application to generate the HTML file. The following graphics made by the project team shows that in an architecture diagram (get the full graphics here, or click to enlarge).