Robot Guitar - bringing unseen data sources into the real world

          Why robot guitar? some back story

In one of those moments when something, some image or concept can completely capture one's imagination, as I idly wandered through the Japanese pavilion at World Expo 1988 in my hometown of Brisbane Australia, I saw a robot playing a classical guitar. And I was awestruck. In fact I still am. I mean, how clever are humans to create a system that plays with such high fidelity, and be perfect in doing so every single time? How?

That robot was really really cool. Even if it is true that a human creator put together the hardware and software in the right order to make it work. But it got me thinking. Was the robot just a tool, I wondered, like a simple guitar plectrum that a human has created and simply told what to do? Was it more than this, was it less than this? I was a teenager in 1988 and with the certainty of a teenager, I resolved then and there that both human AND robot guitar players were cool in subtle and perhaps not-so-subtle ways and yes, I promised myself that I'd have a go at building something like this one day.

As I learnt the guitar slowly over the years I often thought of this robot and wondered how it must compare to the infinitesimally more complex and unknown nuances that humans possess & invest into making music. Surely it would never come close to playing music with the unconscious & sometimes random, very human 'things' that we do. Spontaneous things, actions such as muting certain strings a tiny bit, or bending strings ever so slightly, or selectively just going full whack at it like my guitar hero Billy Bragg does. Maybe it's all those little imperfections, the finger squeaks and educated mistakes that make an interesting piece of music - the mistakes that only a human can make properly.

Many years and much existential ponderings passed in this way, at the same time systems such as the Arduino and Raspberry Pi became prevalent, and electronics became more accessible than ever before. As a hobby, I began rediscovering my basic electronics knowledge, and as my confidence grew I realised that all this stuff was exactly what I needed to build that robot guitar I'd promised myself at Expo88. The elements were now coming together, as I'd always been more than a little hooked on the notion of bringing previously unseen and esoteric data sources into the real world in new and interesting ways. Then in 2014 I went to my first SIGGRAPH conference in Vancouver and was subsequently blown away by the creativity I found, I saw there that people were doing some very cool things and just...putting themselves out there. Just like that! I wish I'd found that conference years earlier.

Now, 27 years on from Expo88, all the ingredients are in place for the world's first "Network Time Protocol accurate, Internet of Things connected, Big Ben chiming grandfather clock, firewall log playing, door chiming, guitar playing robot" thingy, all built by me!. Many were the busy nights and more than a few weekends spent working on this on-again off-again project, hacking together elements of music, computing, engineering, hardware, circuitry, art, lighting, cyber security, time, and working with all sorts of materials. I had to bring it all together.

For those who find all this as interesting as I do, I'll briefly describe the journey, from an account of early prototypes such as the Firewall Tambourine - which was a pivotal stage towards bridging that gap between realtime data and realworld.

          Design notes, final version

As it happens, it's the final stage of the project where the two most important parts of the whole finally come together, these being the servo rig and the coding.

To begin, the servo rig needs to be just right. It has to be stiff enough so as not to move, whilst still allowing the pick to be adjusted up, down and sideways so as to position the tip of the pick just so; critically plucking just enough string whilst remaining at the right angle & depth so as to stay clear & not get stuck - or it would never pluck the string at all.

The picks themselves are real guitar picks which are bent so the pick point is parallel to the strings, with the relatively soft, rather than the harder picks being better for plucking the strings. The picks are affixed to the servo horns by a screw at the pivot and a small pin made from a piece of copper (from a cat6 data cable), which when inserted through the end of the horn and the pick, acts like a staple, thus making the servo and the pick one element. The servos are siliconed to small aluminium angles with slotted holes filed in the top to allow adjustment.

Another important part is the control architecture and getting the Pi and Arduino co-ordinating as intended, with each of these devices being better suited for certain tasks. For example, the Arduino is better at doing proper PWM (Pulse Width Modulation) which is required to control the position of servos, whilst the Pi offered a familiar Ubuntu/Python programming environment, wireless access for NTP and ssh, additional GPIOs and the flexibility to develop the code. The Pi is also used as a platform to write the Processing sketch which is then uploaded to the Arduino itself via the command line Arduino IDE 'ino'. Communication between the Pi and the Arduino is done via a serial connection.

One of the interesting tasks in the coding was that the system needs to keep track of the state of each pick (i.e. its location, is it to the left or right of the string) so it knows in which direction to move the servo. This element is programmed on the Arduino, which sets up all the picks at the initial state when the system initially boots, and then operates whichever servo the Pi asks it to via the serial connection, whilst (importantly) also updating a "pick state" array, storing the current position of each of the six servos, thus keeping state. This part took some coding and testing, but once done it was set and forget.

Other design aspects include LED lighting to illuminate the white picks with blue light. I used the waterclear type LED's instead of the diffused type because I wanted beams of light to really illuminate the picks. I glued the LEDs under the breadboard and also underneath the Pi to give a nice glowing effect with no direct light source visible. At the last minute I also drilled a small hole in the Perspex slab the Arduino sits on and inserted an LED to see what it would look like, and I think it looks real nice!

Altogether there are six yellow manual buttons to operate and tune each pick up and down individually, and two more buttons to run pre-programmed routines. The six tuning buttons are monitored by the Arduino and the other two pre-programmed buttons are monitored by the Pi.

The main frame is made from stained pine, clamped together around the body of the guitar by means of threaded rods, while the foam padding serves to further protect the guitar. Furthermore, in terms of visual appeal I wanted the wiring, circuitry and mechanics to be raw, fairly tidy but definitely not hidden, so the viewer appreciates what goes on behind the scenes.

Finally, as my nephew rightly points out, the frets are in the substantial need of some robotic attention. I agree, however this is enough for now and yes, another maker project is added to the list...


All together there are over 460 lines of code: Python and a small amount of bash for the Raspberry Pi, and Processing sketches for the Arduino. There are all sorts of interesting edge cases and logic that are coded up e.g. did you really want this thing to chime in the middle of the night, what about on weekend mornings, hmmmm??

As I coded, I was constantly reminded how challenging it is to build a machine to do this most trivial task, the act of strumming a guitar. In fact in reading the backstory, I'm struck that this is actually the very concept I wanted to explore in the first place.

Arduino sketch:
Servo control and state awareness

Raspberry Pi python scripts:
     Grandfather clock / Big Ben hourly chiming
     Firewall logging
     Preprogrammed button monitoring (green button)
     Strum down
     Strum up
     Count in, and play a nice tune
     Turn LED's on

          The main rig

          Prototype 2 - Enter the servo

It was clear I needed to move on from the vibrating disks to get some more power to pluck the strings with volume. I bought some micro servos and learned how to control them with Pulse Width Modulation, which was easier than I imagined. Servo control is a task well suited to the Arduino, as it can control them more accurately than the Pi.

The next challenge was fitting the servos in a rig, and building a frame for the rig, Arduino, Pi and breadboard to sit on, all without damaging the guitar. This stage was a big hurdle and the one I spent most time on as I struggled with a myriad of ways to affix all the components. For example at one point I carved the servo rig from a piece of pine, however it was difficult to screw the tiny servo mounting screws into place and allowed for no adjustment. I later discovered aluminium was the perfect material for the servo rig. Pictured is one version of the frame which attached near the base of the guitar. It looked a little like the tailpiece of an old style guitar which I really liked, but it wouldn't fit everything and wasn't adjustable enough.

I also cut picks from all sorts of materials including from the lids of takeaway containers, which were about the right stiffness but they lacked consistency and the plastic was brittle if not cut perfectly.

          Prototype 1 : Vibrating disks on guitar strings

This prototype extends the proof of concept by adding a guitar and more vibrating disks. The disk's wires were soldered to the pairs of cat6 data cables, which then went back to the Pi's GPIO's via the breadboard. This is shown partially completed in the photos above.

Once again the length of the wires was an important factor in maximizing the movement of the disks. I also made some bridges out of Perspex to hold the strings at the resonant lengths at fret 5 and 12, which created a nice harmonic ringing sound, similar to a bell.

The problem with this prototype was volume, it sounded nice but it wasn't very loud because the disks didn't pluck the strings so much as vibrate on top of them; there just wasn't a big enough sound. Also, the effect of the bridges - while very cool and bell-like - lessened the volume significantly.

          Prototype 0 - The PoC firewall tambourine

As a Proof Of Concept I wired up a vibrating disk (like the one you have in your mobile phone that makes it vibrate) into a Raspberry Pi GPIO pin. The vibrating disk sat just above the drum skin and vibrated when it received a current. The Pi was connected to the Internet and a python script tailed the firewall logs so that when the Pi was hit from an IP, the firewall blocked the connection and a current was sent to the vibrating disk, which then jumped up and down on the tambourine. Also at this point, the blue and red LED's strobed in sequence and a pair of piezo speakers sounded in turn. Think police car...

The key in getting the disks to work to maximum effect was in adjusting the bend and stiffness of the connecting wires just so, putting the vibrating disks into resonance and thereby get that thumping on the tambourine. I tested this prototype by scanning it with nmap. In all my career in cyber security I hadn't ever run an nmap against a tambourine before this point, much less did I think I'd later nmap a guitar as well!

As novel as it was to hear the Internet play a tambourine, after a short while I needed to install a small switch on the breadboard to turn the vibrating disk circuit on and off...

Whilst this prototype was only simple, it was a significant 'hello world' moment, as the gap had been bridged between an esoteric realtime data source and making this really, actually, do something interesting in the real world.
Copyright dataviz Australia 2010