What I’ve Been Working On Lately – Recap

I didn’t write for a while1. And it’s not that nothing had happened. The opposite… so much learning and new experiences that I didn’t find the time to log. No, it’s not a lack of time, but rather not internalizing how important it is to stop, asses and capture what I’m learning as I go.

But it’s better done late than never2. So here’s a list of projects I’ve been working on, in no particular order, followed by the list of new skills I’ve learned.

Projects

Outbrain News Brief for Alexa

This is a simple skill for Alexa, that reads summaries of top news stories. So as a user, you have to add this skill to Alexa, and can then ask Alexa “what’s in the news”. Alexa then calls a web service, which I developed. This web-service calls Outbrain and ask for the latest news (using the Sphere platform). It then sends the articles it gets from Outbrain to a summation service (Agolo), and returns them back to Alexa, which then reads the summaries to the user.

Outbrain skill for Alexa

Similar to the skill above3, but with more functionality. This is actually the initial stage of a conversational experience, where users will be able to interact with Alexa to get personalized news stories. So users will be able to guide Alexa through conversation to article from site, or on topic they are interested in, or discover new content based on their interest graph. Here’s a simple sequence diagram that illustrating the current user flow:

sequence-diagram.jpg

My Clipboard

alexa-clipboard-icon.png

Now that’s where things become more interesting, working on my stuff… This is a skill for Alexa that serves as your clipboard. You can say “Alexa, ask my clipboard to remember 212 322 4432” and she’ll remember this phone number for you. “Alexa ask my clipboard what’s in my clipboard” (yeah, redundant, I know…) and she’ll repeat the phone number for you.

Why is it helpful? imaging that you’re on the phone and can’t take a note, or fine a pen to write one down… let Alexa handling it for you… But if you think about a smarter clipboard, one that takes keys and values, you can do much more interesting stuff. For example, ask Alexa to remember that you put your passport in the top drawer. Later on, you can ask her where did I put the passport. But that’s a longer term functionality… I first need to finish the current iteration and get it public (it’s not at the time of writing this…).

Baby Weatr

Artboard_1v2.png

Baby Weatr is a Facebook Messenger app4 that helps parents decide how to dress their kids appropriately for the weather. Well, it designed around my lack of any skill to translate weather into baby wear. So to make sure I’m not endangering our daughter, I decided to build this decision support app.

I’m working on it together with a friend, but this was an opportunity to tie together a lot of the things that I like, and always wanted to use more, such as coding and design5. Initially we tried to outsource the design work, but working with chip freelancers produced deliverable at the quality we paid for, meaning bad. On the other hand, hiring capable designer is expensive. So, I decided to hone on the opportunity to connect with the right side of my brain, and design the first version of the app myself.

Baby Wear is live on Messenger now, so if you need help dressing up your baby – I would love to get your feedback…

Try Baby Weatr

Dlog

While working on the projects above, I did quite a bit of coding. What’s more, this time I coded almost professionally (some of  what I built is going to be used by my company…).

I found that I need to log what I’m doing, so I can backtrack if needed, and won’t make the same mistakes twice. I found that it also accelerated my learning (similar to how writing does…). Git commit, or inline commenting weren’t enough, as I wanted to capture not only the outcome of my thinking, research, trail and error and refactoring. But rather, I wanted to put capture my deliberations and place bread crumbs as I go. I wanted to be able to read back and understand why I made certain decisions. For example, why I selected one data structure and not another, how do I start a flask project, and how do I run a flask app and make it reload every time I make a change.

And so I started to maintain a file called development log, or dlog. I keep it open as part of my workspace and include it in my git repository. Here’s an example of how it looks like (the dlog is in the bottom right quadrant):

I thought that might be something that other developers find helpful, and put it on a separate blog (here). I’m contemplating with the idea of opening this blog for others, with the assumption that if many developers log their process, it will serve as a new form of knowledge repository, stack-overflow extension, or companion to readme documentation.

Things I’ve learned

Chatbots Messenger Apps

Well that’s not new for me… my team is dedicated to messaging apps for awhile now. I think I mentioned before that we’re responsible for the CNN app on Messenger and Kik, as well as for the apps of other notable publishers. What I did learn is how to view these type of apps as the best way to develop an MVP, and how you can build a full experience with building blocks, and minimum amount of code, or back-end services.

I’m used Chatfuel6 as the content management system for the Baby Weatr app, and love the way I can control the behavior of the app, and build it to match the way I’m thinking about flows. Here’s how the Baby Weatr app looks like within Chatfuel: chatfuel-baby-weatr.png

Assistant devices

Assisted device are the conversational version of messenger apps. Here, a user can interact with a device with voice, rather than with text. I’ve been working with Alexa on the skills I mentioned above. I also experimented with Google Home, and their api.ai platform.

I think that these experiences are the real revolution in AI and conversational design, and messaging apps, or chatbots are just a stop in the way. I suspect that FB is going to kill their (less than) year old platform, and bet on live video, VR and maybe voice recognition. Right now the messenger apps are like a ghost town, Much more to say about that, and about what messenger apps are good for (hint: MVP). I’ll keep that to another post.

Python

Python isn’t new to me. I use it occasionally to write scripts to streamline my workflows, or automate tedious manual work. (Automate the Boring Stuff with Python was the book that got me started with python. Highly recommended.)

But this is the first time I’ve used python for real products and services. Using it more intensely I’ve learned how friendly the language is, and how well it fits the way I think about code. I wrote so much, that even google took note, and invited my to the google coding challenge7, mistaking me with a real developer :-). google-code-challange.jpg

Flask

That’s the backbone of almost every one of the projects I listed above. Flask, and it’s Alexa extension – Flask-Ask, are super easy and intuitive packages that help creating web services. I created a template (TODO: push this template to GitHub) and I use it as a starting point with new projects.

Design

Now, that were my passion is at these days. I’ve just finished a 12h Illustrator course in Udemy, and in the middle of a… Illustrator 4 Fashion class. All I think about are shapes and colors, and how I can make them in Adobe Illustrator, my hands are glued to the new Intous Pro I’ve just bought.

In a way, I’m where I wanted to be when I did my bachelor degree – code and design (I graduated as a software engineer, with focus on machine learning and… graphic design).

But in my journey to Illustrator I actually made two stopped, in Inkscape and Sketch. I started with Inkscape, which is great. It’s easy to learn and very powerful. What I like most with Inkscape is the control over the creation and modification of paths, which is way easier and more intuitive than Sketch and Illustrator. I did most of the clothing items to Baby Weatr using it, and posted samples of these designs in a previous post.

But Inkscape lacks in layout and layer management. I also missed smart guides, which makes the interface design much controllable. And so I’ve started to learn Sketch.

I love Sketch’s workflow, as well as the way it lets me organize design assets alongside my artboards. But, it’s not a replacement to Inkscape when it comes to actual illustrations. What I ended up doing is creating the cloth items’ illustrations in Inkscape, and importing them into Sketch, where I did the layout created the sets of outfits.

Here’s how the Baby Weatr project looks like in Sketch: sketch-outfits-page.png sketch-cloth-items.png

And then, as I drawn deeper and deeper into design, I’ve started to learn more about Adobe Illustrator. I tried it while using Inkscape and Sketch, but it seemed too complex, and inaccessible to me. But the more complex it seemed, the more attracted to it I became (no wonder I use Emacs…). When I finished all the clothing sets I needed for the beta launch of Baby Weatr, I decided to get serious and learn Illustrator. After all, it is the tool for designers…

And as I mentioned, 150 episodes later that took 15 hours and span over 3 course, I’m at a point where I feel comfortable with the tool, and starting to do art and design work in it.

Phooo, there was a lot of catching up I needed to do… but it feels good to look at that list and appreciate all the things that I had the chance to learn and experiment with.

Footnotes:

1

And now, just to make sure this post is going to get published, I used one of my hacks, and put it on scheduled publishing…

2

And no, there’s no new year resolution involved in this writing. I don’t like this practice, and don’t set those resolutions…

3

This skill is still in development, so not public and can’t be added to Alexa yet.

4

aka chatbot, but I denounce the term, because it’s lame…

5

I graduated as a software engineer, with a focus on machine learning and… graphic design.

6

They were actually a fierce competitor when we tried to get the CNN project 🙂

7

I completed several stages, but didn’t go all the way, because I had other things to work on, and I’m not going to make a career switch…

Uploading Images To Amazon S3

Here’s a cool little workflow that I use to automate the usage of images in my posts1.

The flow

  • Move the image I would like to use to my black-hole folder
  • When a new file is added to the folder, or existing file is modified, a Hazel is triggered
  • This Hazel rule fires up a python script
  • The python script:
    • Pushes updates to my Amazon S3
    • Returns a link to the uploaded file
    • Place the link in the clipboard
    • Add a log entry that includes the timestemp and the link to that file

Continue reading Uploading Images To Amazon S3

A Recipe To Work-In-Progress Documents

I recently stopped using Evernote and started to manage my notes exclusively in Dropbox. My configuration revolves around a Notes folder. I use nvAlt to browse through the notes in that folder and add new ones. If I want to do more than just a scribble, I use the command-e key binding in nvAlt to open the document in MultiMarkdown Composer.

Storing all my notes in one folder has a major limitation, though. As notes accumulate, looking for a specific note becomes impossible. This is actually one of the main reasons to my departure from Evernote. To avoid this problem, I set Hazel to monitor my Notes folder and move everything that wasn’t modified in the last 30 days to a designated archive folder. Archived notes don’t show in nvAlt, yet easily accessible through Finder.

Now that I have a home to my notes, I would like to add some logic to streamline my writing workflow. To begin with, I would like to aggregate documents I’m working on, and are in other folders, to my main notes’ repository.

For example, I’m currently writing a readme file for one of my git repositories. This repo lives within its own folder, where the readme file resides as well. Keeping this file out of my Notes folder means that it’s a hassle to go back and open it when needed. It also means that I can’t work on it when I’m on my iPhone 1.

So, what I needed was a way to mark a document, and have it magically show up in my Notes folder, hence available in nvAlt. Following is the recipe I came up with to address this need.

Let’s start with the ingredients:

  • Finder
  • Hazel
  • Python

And here’s how to mix these components together:

  1. Open Finder and tag wip the document I want to work on and make available in nvAlt.tag_wip.png
  2. Configure a Hazel rule that monitors my home folder, looking for files containing the wip tag 2. hazel_3.png
  3. Create a python script that takes a file’s path as an input and place a symbolic link to it in my Notes folder.
#! /usr/local/bin/python3

import os, sys, shutil
import logging

# Configuring logging to be written into a file in the system's log folder
logging.disable(logging.CRITICAL)
logging.basicConfig(filename='/Users/ygilad/Library/Logs/Python/myPythonLogs.log', level=logging.DEBUG, format=' %(asctime)s - %(levelname)s - %(message)s')

def moveFileToNote(filePath):
    # Set the link name to the original file.
    # Path to the original file is included for two reasons
    # 1) Avoid naming conflicts and
    # 2) remind myself where this file came from
    fileName = 'link' + filePath.replace('/','_').lower()
    logging.debug('Filename: ' + fileName)

    # Make sure that the input is a file and not a folder
    if len(fileName) > 0:
        try:
            # Add the link to my central note repository
            os.symlink(filePath , '/Users/ygilad/Dropbox/Notes/link-'+ fileName)
            logging.debug('Created a file link')
        except FileExistsError:
            logging.debug('File already exists at the target folder')
    else:
        logging.debug('Input is not a file')

# Accept the path coming from Hazel
hazelLocalFile = sys.argv[1]
logging.debug(hazelLocalFile)

# The body of the script
moveFileToNote(hazelLocalFile)

There is one drawback I wasn’t able to solve – nvAlt doesn’t show the content of the link. All it does show is the path of the original document.nvAlt_and_linked_files.png

While I can’t edit the file directly in nvAlt, I can still do it in MultiMarkdown Composer or Editorial on my iPhone.

Footnotes:

1

I keep git repositories in a local folder out of Dropbox reach, because I heard that you shouldn’t mix the two together.

2

I found that creating a rule that monitors a folder and its sub-folders is a bit tricky, but eventually learned how to do it thanks to this post.

Find Repetitive Words Using Python

Read this question from stackoverflow:

Paris in the
the spring. Not that
that is related.

Why are you laughing? Are my my regular expressions THAT bad??

Have you notice the repetitions? chances are you haven’t. The eye sees what the eye wants to see, and it’ll take away any obstacle to let your brain comprehend. I too often catch myself writing the same word twice. The problem is that when I do, it’s usually too late. The email was sent or post got published.

To make sure I find those repetitions in time, I wrote a simple Python script that removes superfluous spaces and highlight words’ duplications, using CriticMarkup. I run this script as soon as I finish writing. It works much better than my eyes in finding those elusive duplications.

Here’s the script:

#! /usr/local/bin/python3
# removeRepeatWords.py - find and remove repeat words
import logging
logging.basicConfig(filename='/Users/ygilad/Library/Logs/Python/myPythonLogs.log', level=logging.DEBUG, format=' %(asctime)s - %(levelname)s - %(message)s')

logging.disable(logging.CRITICAL)

import pyperclip, re

text = str(pyperclip.paste())

#regex definitions for reapeated spaces
repeatSpacesRegex = re.compile(r'\b(\s)+\1+\b') 

#regex definitions for reapeated words
repeatWordsRegex = re.compile(r'\b(\w+)\b[\s\r\n]*(\1[\s\r\n])+', re.IGNORECASE|re.DOTALL)

#remove the extra spaces
repeatSpces = repeatSpacesRegex.findall(text)

if len(repeatSpces) > 1:
    text = repeatSpacesRegex.sub(r'\1', text)
    print(str(len(repeatSpces)) + ' repeat spaces were removed.')

#remove repeated words
repeatWords = repeatWordsRegex.findall(text)
logging.debug(repeatWords)

if len(repeatWords) > 0:
    text = repeatWordsRegex.sub(r'{~~\1 \2~>\1 ~~}{>>repeating words<<}', text)

pyperclip.copy(text)

To use it, copy the text you want to check into the clipboard. You then run the script and its output will be ready for you back in the clipboard. Just past it over the original text. Note that if the script finds repetitions it won’t remove them, but mark them using CriticMarkup. If your editor supports CM, you can decide whether to accept or reject those changes.

Running this script on the quote from stackoverflow above produces the following output:

Paris in {~~the  the ~>the~~}{>>repeating words<<}spring. Not {~~that  that ~>that~~}{>>repeating words<<}is related.

Why are you laughing? Are {~~my  my ~>my~~}{>>repeating words<<}regular expressions THAT bad??