Wrong Side of Memphis

The Many Challenges of a (Software) Architect

Software Architecture is a fascinating subject. The objective of a solid architecture for a system is to generate an underlying structure that is separates components in a meaningful way, but at the same time, is flexible to allow the system to grow in capacities and functionalities. That it’s performant, reliable and scalable within the required parameters, but it’s as easy to work with as possible.

Who doesn’t like to draw diagrams on whiteboards or glass? Photo by Nataliya Vaitkevich on Pexels.com

Even worse than that, the work is to do that at the same time that a myriad of other competing priorities are being done, new functionalities are added and costs kept under control.

And, of course, while being as simple and elegant as possible.

In essence, nothing new to anyone doing software development…

After all, any program, even very small ones, have certain architecture in their design. Even when tiny, any structure in place can be rightfully understood as the architecture of the system.

Spaghetti is architecture as well. Not a great one, though. Photo by Markus Spiske on Pexels.com

The main difference when we talk about Software Architecture, is that usually we refer to systems that have already certain size, so multiple people are working on them. The view changes from “how much can fit in the head of one person” to “how the work can be divided so it can be done effectively”. A single person can have a strange way of doing things, but it makes sense for them.

And that’s the crucial element, the involvement of different people, and more commonly, different teams in the process. No longer esoteric divisions that make sense only for their creator are a good idea.

I’ve been involved into a lot of architecture discussions and processes in the last 3 to 4 years. First, as part of a group that was trying to give technical direction across the company where I was, and in my current role as Architect, where is a big part of my daily duties.

And it’s almost all about the communication. For example:

  • A very difficult task is to transmit the vision in your head or design to different people that may have other priorities and perhaps they don’t see things in the same way. Most of the time is because they don’t perceive the improvement in the same way, as they are focused in other tasks.
    This is similar to trying to teach someone to use a shortcut to copy/paste instead of going to the menu and select it. Sure, it’s faster, but they don’t see the point because they are focused in writing and not in learning how to be faster. A lot of architectural decisions are about making changes for improvements that may not be immediately obvious, like making processes more resilient.
  • It’s very common to be between two different teams trying to make them cooperate. But this is easier said than done, because each team have their own agenda.
    Another important detail is that there’s no hierarchical relation, so at best you can only influence the teams and that’s a difficult skill.
  • It’s great to have something designed, point a finger in the air and say: “Make It So”, but you need to do follow up on what’s the progress on the different tasks and ensure that the actual implementation of it makes sense and achieves the objectives.
  • There’s also the need to be self-reflective and acknowledge that every step needs to be evaluated critically to be sure that it’s in the right direction.
    The objective of work in the Architecture of the system is not to mandate top-down a Perfect Design™, no matter what. Instead, is to help the different teams into work better. There needs to be feedback in the process.

Surprise, it’s all dealing with people, it’s all soft-skills! Influence, teaching, negotiation, communication… While the tech skills and experience are obviously a prerequisite, in the day-to-day the muscles that gets more flexed are the the soft-skills ones.

Soft skills are hard. Photo by Yan Krukov on Pexels.com

A consequence of this is that pressure and struggle manifest in a different way, as you are not only depending on something that a machine can do, as typical for engineers, but on what other people can do.

It’s a fascinating work, though. I like it a lot as, when everything comes together is great to feel that you are helping a lot of teams to work better together and to make your vision on what direction the system should follow gets implemented.

I like Software Architecture so much that I wrote one book exclusively about it, plus another that also talks a lot about these questions.

And you also get to draw a lot of diagrams! Diagrams everywhere!

Futures and easy parallelisation

Here is one small snippet that it’s very useful when dealing with quick scripts that perform slow tasks and can benefit from running in parallel.

Enter the Futures

Futures are a convenient abstraction in Python for running tasks in the background.

There are also asyncio Futures that can be used with asyncio loops, which work in a similar way, but require an asyncio loop. The Futures we are talking in this post work over regular multitask: threads or processes.

The operation is simple, you create an Executor object and submit a task, defined as a callable (a function) and its arguments. This immediately returns you a Future object, which can be used to check whether the task is in progress or finished, and if it’s finished, the result (the returned value of the function) can be retrieved.

Executors could be based in Threads or Processes.

The snippet

Ok, that’s all great, but let’s go to see the code, which is quite easy. Typically, in your code you will have a list of tasks that you want to execute in parallel. Probably you tried sequentially first, but it takes a while. Running them in parallel is easy following this pattern.

from concurrent.futures import ThreadPoolExecutor

NUM_WORKERS = 5
executor = ThreadPoolExecutor(NUM_WORKERS)


def function(x):
    return x ** 2


arguments = [[2], [3], [4], [5], [6]]

futures_array = [executor.submit(function, *arg) for arg in arguments]
result = [future.result() for future in futures_array]
print(result)

The initial block creates a ThreadPoolExecutor with 5 workers. The number can be tweaked depending on the system. For slow I/O operations like calling external URLs, this number could easily be 10 or 20.

Then, we wrap our task into a function that returns the desired result. In this case, it receives a number and returns it to the power of two.

We prepare the arguments, like in this case the numbers to calculate. Note that each element needs to be a tuple or list, as it will be passed to the submit method of the executor with *arg.

Finally, the juicy bit. We create a futures_array submitting all the information to the executor. This returns immediately with all the future objects.

Next, we call the .result() method on each future, retrieving the result. Note that the result method is blocking, so it won’t continue until all tasks are done.

Et voilà! The results are run in parallel in 5 workers!

A more realistic scenario

Sure, it will be strange to calculate squared numbers in Python in that way. But here is a sightly more common scenario for this kind of parallel operation: Retrieve web addresses.

from concurrent.futures import ThreadPoolExecutor

import requests
from urllib.parse import urljoin


NUM_WORKERS = 2
executor = ThreadPoolExecutor(NUM_WORKERS)

def retrieve(root_url, path):
    url = urljoin(root_url, path)
    print(f'{time.time()} Retrieving {url}')
    result = requests.get(url)
    return result


arguments = [('https://google.com/', 'search'),
             ('https://www.facebook.com/', 'login'),
             ('https://nyt.com/', 'international')]
futures_array = [executor.submit(retrieve, *arg) for arg in arguments]
result = [future.result() for future in futures_array]
print(result)

In this case, our function retrieves a URL based on a root and a path. As you can see, the arguments are, in each case, a tuple with the two values.

The retrieve function joins the root and the path and gets the URL (we use the requests module). The print statement work as logs to see the progress of the tasks.

If you execute the code, you’ll notice that the first two tasks start almost at the same time, while the third takes a little more of time and start around 275ms later. That’s because we created two workers, and while they are busy, the third task needs to wait. All this is handled automatically.

1645112889.005687 Retrieving https://google.com/search
1645112889.006123 Retrieving https://www.facebook.com/login
1645112889.2814898 Retrieving https://nyt.com/international
[<Response [200]>, <Response [200]>, <Response [200]>]

So that’s it! This is a pattern that I use from time to time in small scripts that very handy and gives great results, without having to complicate the code to deal with cumbersome thread classes or similar.

Photo by KEHN HERMANO on Pexels.com

One more thing

You may structure the submit calls with the specific number of parameters, if you prefer

arguments = [('https://google.com/', 'search'),
             ('https://www.facebook.com/', 'login'),
             ('https://nyt.com/', 'international')]
futures_array = [executor.submit(retrieve, root, path) for root, path in arguments]
result = [future.result() for future in futures_array]

The *arg part allows to use a variable number of arguments using defaults depending on the specific call, and allow for fast copy/paste.

“Python Architecture Patterns” now available!

My new book is available!

Now available!

Current software systems can be extremely big and complex, and Software Architecture deals with the design and tweaking of fundamental structures that shape the big picture of those systems.
The book talks about in greater detail about what Software Architecture is, what are its challenges and describes different architecture patterns that can be used when dealing with complex systems. All to create an “Architectural View” and grow the knowledge on how to deal with big software systems.
All the examples in the book are written in Python, but most concepts are language agnostic.

This joins the collection of my books! I’m quite proud to have been able to write four books already…

They look stylish on any tech library

They are all available in Amazon and in the Packt website.

Python Automation Cookbook for only $5!

There’s currently a Christmas offer in Packt website when you can get all ebooks for just $5 or 5€.

It’s a great opportunity to get the second edition of the Python Automation Cookbook and improve your Python skills for this new year!

Full of great recipes!

Also available, of course, is Hands-On for Microservices with Python, for people interesting in learning about Docker, Kubernetes, and how to migrate Monolithic services into Microservices structure.

Look all those microservices!

A great opportunity to increase your tech library!

Still working from home after all those years

We are all experts in working from home now, right?

Since March 2020 we’ve been stuck in this strange situation where time has stopped and we are working regularly from home, at least almost everyone in the software industry. Because we were already a bit ahead of the curve.

I was seeing more and more remote work since at least a few years before. The first time that I had any meaningful remote work was around 2005. Back then, I was working as a consultant and I had regular meetings with customers onsite, carrying a laptop. Somedays I would finish the reports and other work from home, as it didn’t make sense to get back to the office. It was a small company, but most people will have similar experiences, some days not going to the office.
It took a while for me to encounter back in a situation where remote work was common, but in the last half of the 2010 it was more and more common. I knew some people that moved countries and kept their jobs, so they work mostly remotely. And on-call work routinely require people being able to connect from their homes, setting up a VPN, etc.

That was growing over time. Three years ago, it was common where I worked to work from home one or two days of the week, and I’ve been taking advantage of going for a month overseas to Spain and keep working remotely most days.

But now it has taken a sudden acceleration, and we experienced quite a lot of work from home. I’m going just to put in writing some of the elements I think are quite important in this, or at least they are for me.

Space.

While for working sparingly one day or two you can sit on the kitchen table or from the coach, this is hardly a sustainable option. You need a dedicated space that can be used for working effectively day after day.

No matter what TV series tries to tell you, I really need sunlight for work. Photo by Andrew Neel on Pexels.com
Read More

“Python Architecture Patterns” book announced!

We are getting close to the end of the year, but I have great news! A new Python book is on the way, and will be released soon.

Fun with software architecture

Current software systems can be extremely big and complex, and Software Architecture deals with the design and tweaking of fundamental structures that shape the big picture of those systems.
The book talks about in greater detail about what Software Architecture is, what are its challenges and describes different architecture patterns that can be used when dealing with complex systems. All to create an “Architectural View” and grow the knowledge on how to deal with big software systems.
All the examples in the book are written in Python, but most concepts are language agnostic.

This new book is related to my previous one “Hands-On Docker for Microservices with Python“, as it also deals with certain architecture elements, though in the new book it has a broader approach.

With this book, I will have four published technical books, well, three if you count the two editions of the Python Automation Cookbook. I count them as two as the writing effort was there!

I hope that it will be fully available soon, so far it’s only in pre-sale. Stay tuned!

Basic Python for Data Processing Workshop

I’ll be running a workshop at the European ODSC this 8th of June.

Python for Data Processing

The objective of the session is to provide some basic understanding of Python as a language to be used for data processing. Python syntax is very readable and easy to work with, and its rich ecosystem of libraries makes it one of the most popular programming languages in the World.

We will see some common tools and characteristics of Python that are basic to analyze data, like how to import data from files and to generate results in multiple formats. We will also see some ways to speed up the processing of data.

This workshop is aimed at people with little to no knowledge of Python, though some programming knowledge is required, even if it’s in a different language.

Session Outline
Lesson 1: Basic Python. 

Introduction to basic operations with Python that is basic for treating data and creating powerful scripts. Also, we will learn how to create a virtual environment to install third-party libraries and discuss how to search and find powerful libraries. It will also include creating flexible command-line interfaces.

Lesson 2: Dealing with files

Read and write multiple files, from reading CSVs to ingest data to writing HTML reports including graphs with Matplotlib. We will also cover how to create Word files and PDFs.

Lesson 3: Efficient data treatment

Learn how to deal with data in an efficient manner in Python, from using the right data types, to use specialized libraries like Pandas, and using tools to run Python code faster.

Background Knowledge
Programming knowledge, even if it’s intro in Python

Got interviewed as PyDev of the Week!

I got interviewed as part of Mike Driscoll’s PyDev of the week series. You can check the interview here

Podcast Wrong Side of Life

I started a new podcast with Sana Khan (@LegalSana_Khan and http://sanakhanwrites.com/), talking about things that interest us, like society stuff, tech, laws, movies… We recorded already a few episodes, so perhaps you want to take a look.

You can listen to it in http://wrongsideoflife.com

It’s also available in Apple Podcasts, Spotify and you can search it directly in common podcast applications like Overcast or Castro.

Colourful iMacs again

I really love the new iMacs. The design looks amazing and it reintroduced something long missing in Apple computers. Vibrant colours

I’ll be waiting to the “big” iMac (the current-27 inches size) which I suspect will be renamed as iMac Pro or iMac Plus to get clarity in the product line.

The rest of introduced details are what you’ll expect on this iteration:

Apple Silicon, still M1 chip. Probably a new chip will be presented in Autumn, likely after the introduction of the new iPhone. The Apple Silicon appears to have two models, one for all the “consumer” products, with the introduced M1 products so far (Mac Mini, iPad Pro, MacBook Air and iMac) and a more powerful, “pro”, option (bigger iMac, MacBook Pro) Mac Pro is a special case that probably will get either multiple “pro” chips or a special, “top” chip.

The possibility of using Touch ID in the wireless keyboard was unavoidable, as it’s a really convenient feature on laptops. Same for the webcam, which was due for upgrade. Next step will be to incorporate Face ID as well.

The new design also points to what to expect in next years. Thinner, lighter design which will remove the chin at some point. Same for bezels, which will become smaller. The addition of an external power adapter is strange, though it moved the Ethernet plug out of the back, and I imagine makes the screen lighter.

I hope that it has the same or similar colours than the introduced ones. because they look amazing… And the matching mice, trackpads and keyboards are lovely

I’m still debating myself whether I like most the purple or orange
%d bloggers like this: