Marks make meaning – ST3Mbot learns to draw

Google Brain have a research project ‘magenta’ that investigates Artificial Intelligence in creativity, and as part of this used the popular “quick draw” online game to train a neural network.
The resulting model: sketch-rnn uses the collected data for a number of scenarios:

  • to predict the subject of a sketch before it is completed,
  • to interpolate the ‘in-betweens’ from one sketch to another
  • and to create a variation on a provided sketch based on the understanding the model has of the subject.

I think this work is fantastic, and has huge potential for building tools and workflows.
The outputs are generally recognisable, if primitive, for example, if you run the variational autoencoder with a cat model, the resulting outputs are recognisable as cats, but maybe not very well drawn cats.

Variational autoencoder output

What’s intriguing is that the training data comes from millions of people of all skills levels, awkwardly  trying to draw a given subject, probably with a mouse, in under 20 seconds.
What would the output be like if the model was trained by folk who actually knew how to draw?

I am immodest enough to consider myself able to draw, and in a previous life as an illustrator actually got paid for drawing cats, quickly.

cat drawing from Collins Educational book

However after a weekend sketching cats I started to realise drawing a 7000 image training set of cats is going to take a fair amount of time. So I when the opportunity to get involved with the University of Brighton “Marks make meaning” event  came along it seemed a good opportunity to gather sketches from folks.

To encourage folks to provide training data (cat drawings)
I’ve reconfigured one of my robots- ST3Mbot who usually acts as an ambassador for STEM Sussex, to try something a little more artistic.

ST3Mbot sketchrnn
I upgraded his internals to generate and print out example sketches, share the #ST3Mbotcats hashtag, and hopefully survive the length of the exhibition- from the 12th to 29th March.

training data
Once I’ve created a ‘better’ training dataset I’ll post the results here. Hopefully if it proves interesting enough I’ll carry on with investigations beyond the realm of just cats.

Leave a Reply