Uploading Images To Amazon S3

Here’s a cool little workflow that I use to automate the usage of images in my posts1.

The flow

  • Move the image I would like to use to my black-hole folder
  • When a new file is added to the folder, or existing file is modified, a Hazel is triggered
  • This Hazel rule fires up a python script
  • The python script:
    • Pushes updates to my Amazon S3
    • Returns a link to the uploaded file
    • Place the link in the clipboard
    • Add a log entry that includes the timestemp and the link to that file

Continue reading Uploading Images To Amazon S3

Developer for a day

Earlier today I demoed a simple search page that I developed as part of a 24 hour hackathon here at Outbrain. What makes this search page unique is that it uses the Sphere platform1 to find content that isn’t only relevant to the search query, but also caters to the user’s interests. So, for example, if I search for “python”, I’ll get articles about python, the programming language. If my mom, on the other hand, do the same search, she’ll get articles like this one.

But what I developed isn’t the subject of this article (I might write about it in a separate post). What I do want to share is my experience at putting a developer hat for a day. As a product manager, who works very closely with engineers, this turned out to be an invaluable lesson.

I had no intention of building anything in that hackathon. My plan was to help and support the hacking teams2. A few minutes after the hackathon began, though, I thought it will be cool to build this search page. Unfortunately, at that point there were no developers around to whom I could pitch my idea. All of them were already assigned to other teams. But I got this impulse to build something, so I decided to challenge my lack of development skills, and form a team of one.

Well, saying that I’m not a developer isn’t entirely true. Afterall, I graduated as a software engineer, I understand technology, I can speak intelligently about software architecture and design patterns, and I was intimately involved in designing and building the Sphere platform. I even do some coding in my spare time3. But, I’ve never coded with a mission or under a strict deadline. So I thought, this would be an opportunity to get serious about coding. Indeed, serious I became, spending the next 24 hours (minus 6 hours of sleep) hacking my way toward something I’ll be proud to present.

Here’s how my next 24 hours have looked like:

10:20am – 20 minutes into the hackathon I’m having this idea, and yada yada yada I decide to code.

10:40am – I can see what the architecture of the solution should look like, and what APIs I’ll have to use. It’s going to be easier than I thought…

11:30am – Hitting a dead-end. My initial approach won’t work, because I don’t have access, from the environment I’m using, to the API I rely on. Need to think of a new direction.

12:00pm – Found a new direction. I’m not sure it’s the right one, and I’ll have to learn a framework that I didn’t use before (well, I didn’t use any framework before…), but I’m running out of options, so I’m taking the chance. I find a tutorial, and hope to learn everything I need to know before the hackathon is over.

1:30pm – Urr… this is the longest tutorial ever, and it’s not even related to the use-case I’m trying to solve.

2:15pm – OK, good news and bad news. The good – I finally finished the tutorial. The bad – I still don’t have a clue how to work with this framework to build what I have in mind.

3:00pm – 5 hours have passed, and I still have nothing. What’s worse – people around me have high expectations of me. I have no idea why, but I know I’m going to disappoint them. I’m loosing my patience, and even the quietest chatter in the room distracts me. I’m agitated, and my fuse shortens by the minute. I hope no one will talk to me. I need silence. Maybe I should put headphones on, or go to a secluded room…

4:10pm – Half a day went by, and I’m farther away from my initial idea than I was 6 hours ago. Maybe some coffee and fresh air will help me regain energy and spirit.

4:20 – I don’t know if it’s the coffee or the time off the computer, but I’m thinking more clearly now. In fact, I have an idea. I need to run back to the office.

5:30pm – I have a working solution! It looks awful, but works…. all I have to do now is take care of the front-end. Easy breezy.

9:15pmI hate CSS. I’m about to loose my mind. I must eat.

10:00pm – I need to have some sleep. I’m sure things will look easier tomorrow.

8:10am – CSS is still CSS.

9:55am – 5 minutes to my demo. With some help, hand-holding and a lot of duct-tape my work is somewhat presentable. I hope people see the potential, and won’t get caught-up with the UI.

10:10am – I’ve just presented my thing. I feel awesome. I built something and got people’s applause.

Now, what did I learn from this schizophrenic experience:

I don’t want to be a developer.

Yeah, as simple as that. I’ll keep doing it as a hobby, but I’ll never do it professionally. I mean, I love the problem solving, and creating something with my own hands is amazing. But, getting sucked into the smallest of details, spending hours trying to figure out what I did wrong, only to find a missing ;, and wasting tons of time on configuration before I can do anything, make me go nuts…

OK, so as this door is now closed, what did I learn about coding that will help me understand engineers better?

Coding requires focus

Soooo much of it. Even the slightest distraction can throw your thought process miles away. It then takes a lot of time to regain your thoughts, and get back to where you were before. I now understand even better Joel’s “Human Task Switches Considered Harmful” post.

Tools are important

Don’t ask why, but I’ve started to learn Emacs recently, and so far I love it. But, Emacs isn’t the most dummy proof app out there. Here’s a funny chart, that actually stops being that funny when you start learning Emacs:

3251176498_c3485a55fb.jpg

So, since I’m somewhere at the bottom of the learning curve still, I thought this hackathon would be an opportunity to learn the tool better and faster. But as soon as time started to press on me, and at the first instance when I didn’t know how to do something in Emacs that was trivial in other tools, I closed it and opened the other tools I feel comfortable with (Sublime and CodeRunner).

Knowing what’s the expected outcome is key

Having a clear idea of what my end product should do, and to some extent – how it should look like, was crucial. I had so much to learn in a very short time, but knowing what my end goal was kept me focused. It also helped me stay on course and learn only what was relevant to getting my project done (otherwise I have the tendency to drift away quickly).

And lastly, I experienced first hand how deadlines and quality [don’t] play [well] together:

Code becomes crappier as deadline approaches

You might think that my experience isn’t a good enough example, and I might agree. I’m just saying that now I can better relate to this #NoDeadlines trend.

If you’re involved, at any capacity, in product development, you already know this lesson, because crappy code keeps bouncing back at you and eats the time and resources you need in order to build new stuff. Don’t be fooled by fancy terms, such as “tech-debt” and “refactoring” – these are politically correct ways to refer to crappy code. And the stricter the deadlines, the more of it you’ll get4.

So with that, I’ll put my developer hat down. It’s too big for me…

Footnotes:

1

Check out the platform here, and leave a comment if you’re interested to learn more.

2

There were 19 internal and external teams hacking.

3

In case you’re interested, here’s my GitHub account.

4

No, it doesn’t mean I won’t ask for time-lines and set deadlines in the future…

Copy Git Changes to a New Branch

I’m still not fluent with git and version control. I manage repositories for projects I’m working on, but sill have hard time managing my changes, commits and branches.

For example, I’m currently working on integrating Emacs’s org-mode support to Pelican, a static web-page generator I use for this blog. I was proud of myself for remembering to create a new branch when starting the work. Somewhere in the middle of the integration, I drift away, and started to explore a new idea – adding a commenting system to the site. Before I knew it I was already working on it. Unfortunately, not only that I didn’t work on that feature in a dedicated branch, I was still on the branch I created for the org integration.

I wondered if there is a way to take the changes I made since the last commit, and pour them over into a new branch. Luckily, there is. Here’s how, thank to this answer in stack-overflow:

You can simply check out a new branch, and then commit:

git checkout -b my_new_branch
git commit

Checking out the new branch will not discard your changes.

Tried it and it worked.

A Recipe To Work-In-Progress Documents

I recently stopped using Evernote and started to manage my notes exclusively in Dropbox. My configuration revolves around a Notes folder. I use nvAlt to browse through the notes in that folder and add new ones. If I want to do more than just a scribble, I use the command-e key binding in nvAlt to open the document in MultiMarkdown Composer.

Storing all my notes in one folder has a major limitation, though. As notes accumulate, looking for a specific note becomes impossible. This is actually one of the main reasons to my departure from Evernote. To avoid this problem, I set Hazel to monitor my Notes folder and move everything that wasn’t modified in the last 30 days to a designated archive folder. Archived notes don’t show in nvAlt, yet easily accessible through Finder.

Now that I have a home to my notes, I would like to add some logic to streamline my writing workflow. To begin with, I would like to aggregate documents I’m working on, and are in other folders, to my main notes’ repository.

For example, I’m currently writing a readme file for one of my git repositories. This repo lives within its own folder, where the readme file resides as well. Keeping this file out of my Notes folder means that it’s a hassle to go back and open it when needed. It also means that I can’t work on it when I’m on my iPhone 1.

So, what I needed was a way to mark a document, and have it magically show up in my Notes folder, hence available in nvAlt. Following is the recipe I came up with to address this need.

Let’s start with the ingredients:

  • Finder
  • Hazel
  • Python

And here’s how to mix these components together:

  1. Open Finder and tag wip the document I want to work on and make available in nvAlt.tag_wip.png
  2. Configure a Hazel rule that monitors my home folder, looking for files containing the wip tag 2. hazel_3.png
  3. Create a python script that takes a file’s path as an input and place a symbolic link to it in my Notes folder.
#! /usr/local/bin/python3

import os, sys, shutil
import logging

# Configuring logging to be written into a file in the system's log folder
logging.disable(logging.CRITICAL)
logging.basicConfig(filename='/Users/ygilad/Library/Logs/Python/myPythonLogs.log', level=logging.DEBUG, format=' %(asctime)s - %(levelname)s - %(message)s')

def moveFileToNote(filePath):
    # Set the link name to the original file.
    # Path to the original file is included for two reasons
    # 1) Avoid naming conflicts and
    # 2) remind myself where this file came from
    fileName = 'link' + filePath.replace('/','_').lower()
    logging.debug('Filename: ' + fileName)

    # Make sure that the input is a file and not a folder
    if len(fileName) > 0:
        try:
            # Add the link to my central note repository
            os.symlink(filePath , '/Users/ygilad/Dropbox/Notes/link-'+ fileName)
            logging.debug('Created a file link')
        except FileExistsError:
            logging.debug('File already exists at the target folder')
    else:
        logging.debug('Input is not a file')

# Accept the path coming from Hazel
hazelLocalFile = sys.argv[1]
logging.debug(hazelLocalFile)

# The body of the script
moveFileToNote(hazelLocalFile)

There is one drawback I wasn’t able to solve – nvAlt doesn’t show the content of the link. All it does show is the path of the original document.nvAlt_and_linked_files.png

While I can’t edit the file directly in nvAlt, I can still do it in MultiMarkdown Composer or Editorial on my iPhone.

Footnotes:

1

I keep git repositories in a local folder out of Dropbox reach, because I heard that you shouldn’t mix the two together.

2

I found that creating a rule that monitors a folder and its sub-folders is a bit tricky, but eventually learned how to do it thanks to this post.

Find Repetitive Words Using Python

Read this question from stackoverflow:

Paris in the
the spring. Not that
that is related.

Why are you laughing? Are my my regular expressions THAT bad??

Have you notice the repetitions? chances are you haven’t. The eye sees what the eye wants to see, and it’ll take away any obstacle to let your brain comprehend. I too often catch myself writing the same word twice. The problem is that when I do, it’s usually too late. The email was sent or post got published.

To make sure I find those repetitions in time, I wrote a simple Python script that removes superfluous spaces and highlight words’ duplications, using CriticMarkup. I run this script as soon as I finish writing. It works much better than my eyes in finding those elusive duplications.

Here’s the script:

#! /usr/local/bin/python3
# removeRepeatWords.py - find and remove repeat words
import logging
logging.basicConfig(filename='/Users/ygilad/Library/Logs/Python/myPythonLogs.log', level=logging.DEBUG, format=' %(asctime)s - %(levelname)s - %(message)s')

logging.disable(logging.CRITICAL)

import pyperclip, re

text = str(pyperclip.paste())

#regex definitions for reapeated spaces
repeatSpacesRegex = re.compile(r'\b(\s)+\1+\b') 

#regex definitions for reapeated words
repeatWordsRegex = re.compile(r'\b(\w+)\b[\s\r\n]*(\1[\s\r\n])+', re.IGNORECASE|re.DOTALL)

#remove the extra spaces
repeatSpces = repeatSpacesRegex.findall(text)

if len(repeatSpces) > 1:
    text = repeatSpacesRegex.sub(r'\1', text)
    print(str(len(repeatSpces)) + ' repeat spaces were removed.')

#remove repeated words
repeatWords = repeatWordsRegex.findall(text)
logging.debug(repeatWords)

if len(repeatWords) > 0:
    text = repeatWordsRegex.sub(r'{~~\1 \2~>\1 ~~}{>>repeating words<<}', text)

pyperclip.copy(text)

To use it, copy the text you want to check into the clipboard. You then run the script and its output will be ready for you back in the clipboard. Just past it over the original text. Note that if the script finds repetitions it won’t remove them, but mark them using CriticMarkup. If your editor supports CM, you can decide whether to accept or reject those changes.

Running this script on the quote from stackoverflow above produces the following output:

Paris in {~~the  the ~>the~~}{>>repeating words<<}spring. Not {~~that  that ~>that~~}{>>repeating words<<}is related.

Why are you laughing? Are {~~my  my ~>my~~}{>>repeating words<<}regular expressions THAT bad??