Problem
#
Some imports in my python code are slow. How can I figure out which ones are the source of slowness?
Solution
#
Python offers a really useful functionality you can use that will list how long each import took. By passing the -X importtime argument to your python command when you execute your script it will print out both the cumulative time (including nested imports) and self time (excluding nested imports) of each import.
python -X importtime your-script.py
Running python -X importtime my-script.py on an empty script returns the following (on Windows 7, Python 3.7.5)
import time: self [us] | cumulative | imported package
import time: 52 | 52 | zipimport
import time: 367 | 367 | _frozen_importlib_external
import time: 55 | 55 | _codecs
import time: 530 | 585 | codecs
import time: 520 | 520 | encodings.aliases
import time: 1107 | 2210 | encodings
import time: 328 | 328 | encodings.utf_8
import time: 39 | 39 | _signal
import time: 357 | 357 | encodings.latin_1
import time: 34 | 34 | _abc
import time: 312 | 345 | abc
import time: 474 | 819 | io
import time: 113 | 113 | _stat
import time: 264 | 377 | stat
import time: 269 | 269 | genericpath
import time: 794 | 1062 | ntpath
import time: 871 | 871 | _collections_abc
import time: 915 | 3223 | os
import time: 490 | 490 | _sitebuiltins
import time: 57 | 57 | _locale
import time: 934 | 991 | _bootlocale
import time: 421 | 421 | encodings.cp1252
import time: 472 | 472 | types
import time: 394 | 394 | warnings
import time: 440 | 834 | importlib
import time: 263 | 263 | importlib.machinery
import time: 554 | 816 | importlib.abc
import time: 64 | 64 | _operator
import time: 792 | 856 | operator
import time: 343 | 343 | keyword
import time: 43 | 43 | _heapq
import time: 405 | 447 | heapq
import time: 85 | 85 | itertools
import time: 328 | 328 | reprlib
import time: 69 | 69 | _collections
import time: 1460 | 3585 | collections
import time: 44 | 44 | _functools
import time: 596 | 640 | functools
import time: 784 | 5008 | contextlib
import time: 651 | 7308 | importlib.util
import time: 1095 | 1095 | pywin32_bootstrap
import time: 231 | 231 | sitecustomize
import time: 10744 | 24972 | site
For a script with a simple import argparse, I get the following output:
import time: self [us] | cumulative | imported package
import time: 70 | 70 | zipimport
import time: 341 | 341 | _frozen_importlib_external
import time: 54 | 54 | _codecs
import time: 457 | 511 | codecs
import time: 456 | 456 | encodings.aliases
import time: 1030 | 1997 | encodings
import time: 215 | 215 | encodings.utf_8
import time: 38 | 38 | _signal
import time: 268 | 268 | encodings.latin_1
import time: 33 | 33 | _abc
import time: 398 | 431 | abc
import time: 311 | 741 | io
import time: 87 | 87 | _stat
import time: 271 | 357 | stat
import time: 196 | 196 | genericpath
import time: 416 | 612 | ntpath
import time: 714 | 714 | _collections_abc
import time: 610 | 2292 | os
import time: 229 | 229 | _sitebuiltins
import time: 48 | 48 | _locale
import time: 246 | 293 | _bootlocale
import time: 217 | 217 | encodings.cp1252
import time: 488 | 488 | types
import time: 279 | 279 | warnings
import time: 461 | 740 | importlib
import time: 269 | 269 | importlib.machinery
import time: 557 | 825 | importlib.abc
import time: 63 | 63 | _operator
import time: 808 | 871 | operator
import time: 336 | 336 | keyword
import time: 41 | 41 | _heapq
import time: 336 | 376 | heapq
import time: 69 | 69 | itertools
import time: 341 | 341 | reprlib
import time: 70 | 70 | _collections
import time: 1136 | 3197 | collections
import time: 69 | 69 | _functools
import time: 642 | 710 | functools
import time: 801 | 4708 | contextlib
import time: 688 | 6959 | importlib.util
import time: 934 | 934 | pywin32_bootstrap
import time: 224 | 224 | sitecustomize
import time: 9323 | 20954 | site
import time: 698 | 698 | enum
import time: 55 | 55 | _sre
import time: 417 | 417 | sre_constants
import time: 372 | 789 | sre_parse
import time: 443 | 1286 | sre_compile
import time: 323 | 323 | copyreg
import time: 716 | 3021 | re
import time: 725 | 725 | locale
import time: 948 | 1673 | gettext
import time: 986 | 5678 | argparse
The package are listed in order that they are resolved. In argparse case, os and sys were already loaded, so it first loads re, then gettext. Once both are loaded, argparse has finished loading.
The way the cumulative column is computed is to take all the prior self that are a level higher than the package you’re looking at. For example (if we take the io package):
import time: 33 | 33 | _abc
import time: 398 | 431 | abc
import time: 311 | 741 | io
311 + 398 + 33 = 742
We can see here that the numbers are not necessarily equal to one another, this might be due to precision used to do the computation while the rendering of numbers is rounded.
Note that the load time of a package may be different depending on which script you load because dependencies of the package may have already been loaded in some cases, while in others it may have to load them.
Looking at text might be your thing, but if you’re more visual, there’s a tool called 
tuna which will consume this output and create an icicle plot you can look at to find which imports are the slowest/longest.
References
#
Question
#
How can you tell if you are a low performer?
Answer
#
I always prefer to compare myself against my prior self and not against others. Thus, I would consider myself a low performer if my throughput is lower than what it has been on average in the past. This may happen for many reasons, amongst them it would be because I’m learning something new, so I’m spending a good chunk of my time on learning and less on executing. It might be because I’m trying different ideas to find the best one because I’m working on something I’ve never worked before.
It’s generally easy for a programmer to tell whether he’s been more or less productive than the prior week. It is mostly based on feelings, where you feel good when you are productive and less good when you’re not making any progress or facing issues.
If you think and feel that you are performing poorly, start recording more thoroughly what you are working on. Identify when you start and finish working on a task, and when you get blocked, write down why. After a few weeks, look at what you wrote and assess what might cause you to feel that you are a low performer. Is it because you’re working on a task you are not good at? Is it because of a lack of motivation on the task you’ve been assigned to?
With more information in hand to determine why you feel that you are a low performer, you will be able to devise a plan so that you can once again feel like a high performer.
Problem
#
I have two test files with the same name and pytest complains. How do I make it work without changing the test filenames?
Example directory structure
/path/to/project/tests
├── a/
│ └── test_a.py
└── b/
└── test_a.py
Error message:
import file mismatch:
imported module 'test_a' has this __file__ attribute:
/path/to/project/tests/a/test_a.py
which is not the same as the test file we want to collect:
/path/to/project/tests/b/test_a.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
Solution
#
Add a __init__.py to each directories with tests files that have the same name. Technically, you only need to have a __init__.py file in one of the two directories, so that one is in a package while the other one is in a different one. Adding it in both simply prevents this issue from occurring again if you were to add a third file test_a.py.
/path/to/project/tests
├── a/
│ ├── __init__.py
│ └── test_a.py
└── b/
├── __init__.py
└── test_a.py
Problem
#
I use pytest with coverage and I want to see the files that have no coverage.
Solution
#
It appears that pytest and pytest-cov will not list someof the files that are under namespace packages, while it will work fine for files in regular packages (see 
PEP 420 on the topic of implicit namespace packages).
To fix this problem, one solution is to add __init__.py files in all of your directories in order to create regular packages.
If you are using PyCharm Professional, you can simply run your test with coverage. This will allow you to identify all the files that have currently no coverage as they will appear with coverage = 0%.
References
#
Question
#
What should be defined to make a user demo walkthrough successful?
Answer
#
You need to define what you want to learn from the demo walkthrough: where does the user ask questions? where does he stay stuck? what is easy/hard for him to do? what does he think about when he goes through the demo? what is/isn’t working? what frustrates him? where does the user want to have more guidance?
The user doing the walkthrough should be as close as possible to the ideal user otherwise you may get feedback that is biased on their own experience. A user with too much knowledge compared to your target user will be able to do many things your target user may need help with and they may assume a lot of things because they know about them. On the other hand, a user with too little knowledge will require help in many places where the target user is expected to have knowledge, which may make the demo walkthrough slower than desired.
The walkthrough should have a clear scenario. You may only give an initial setup to the user and a desired goal and let them figure everything out by themselves. You may also go with a more directed approach, where you tell them what to do and you see if the instructions are clear enough to accomplish the steps. The first approach is interesting because it allows you to observe variability in how to solve a problem.
Question
#
How can you tell when you’re working on the wrong task?
Answer
#
You may be working on the wrong task because priorities have changed. To determine if that is the case, you should ask yourself whether completing the task provides value, either to you or your users. If the answer is no, drop the task. If the answer is yes, you should determine if it is the utmost important task at the moment. If the answer is no, then figure out which task is. If the answer is yes, then proceed.
You may be working on the wrong task because you don’t have the necessary information to complete the time in an appropriate amount of time. If you find yourself spending most of your time gathering information instead of accomplishing the task that should be done with the information you need, then it may not be the right time to do the task yet. You may have to create a prior task which is to acquire the necessary knowledge to execute the original task.
If you notice that to complete your task there are pre-requisites that should have been completed, then you should work on those instead of the task with those dependencies. In some cases you may realize that you can’t accomplish a task because you don’t have the tooling necessary or the technology to accomplish the task is not available yet.
As I suggest in my article Given that you define a ROI on a task, when should you stop working on a task and abandon it given its cost?, you should estimate how long you expect for a task to take. At the half time, you should evaluate whether you’ll be able to complete the task by the estimate’s deadline. If you can’t, then you should either drop the task (if you can), or look for alternative ways to get the task completed, either by asking a more experienced person or by simplifying the task.
References
#
Problem
#
My pytests take a while to complete, how can I speed up the process?
Solution
#
A fairly cheap solution is to use parallelization to run your tests on multiple CPUs instead of the 1 cpu used by default. To do so, you can install pytest-xdist. Once the extension is specified, all you need to do is add -n auto when you call pytest.
Another thing you should do that requires more effort is to investigate which of your tests are consuming a lot of CPU time to execute. To do so, use the --durations=0 flag when you call pytest. A report will be generated after your tests have run that lists how long setting up, running and tearing down each specific test took. The list is ordered from longest to shortest durations, meaning that the tests that have the most potential for being optimized will be at the top. You should focus on these tests because the longest one will determine how long it would take to run your tests even if you had an infinite amount of CPU cores.
Investigate why certain tests take a while to execute.
- Are some tests computing something that takes a while and is computed exactly the same way by multiple tests? Precompute this result once and share it between the different tests (think of it as a fixture).
- Are calls to a slow external API done? If you are not testing that the remote API is changing, store example responses and emulate receiving them.
- Is there a loop in the test that runs hundreds of thousands iterations while the same test could be executed with only a thousand iterations?
References
#
Problem
#
I want to include mypy as part of my CI pipeline but my existing code contains a lot (> 100, but < 500) of issues. How can I get started?
Solution
#
Create a minimalist configuration of mypy such that it will list issues that need to be fixed and return a non-zero exit code. Based on the problem definition, we assume that at this step you have more than 100 issues that are listed and that fixing those issues will take many hours you’d rather invest in improving the code than to fix typing issues.
Add a step in your CI pipeline that runs mypy and list all those issues. Verify that it indeed breaks the build.
Once you’ve satisfied yourself that CI fails, we will “fix” the mypy issues by adding the #type: ignore and/or # noqa comment after the offending lines with issues. This will have the effect of resolving all the currently found mypy issues, such that mypy should now return a zero exit code. With this, any future code that fails to pass the mypy check will break the build. This will allow you to use mypy from this point forward to check your types.
I suggest adding an additional comment such as # FIXME: TICKET-ID, where TICKET-ID refers to the id of a ticket in your issue tracking system that explains that you need to take care of this technical debt.
Always prefer to fix the issues instead of ignoring them. However, also consider whether fixing those issues is an appropriate use of your time when you want to introduce mypy (which should be as soon as possible in my opinion).
References
#
Question
#
What should I know about my users if I want to build appropriate software for them?
Answer
#
If you are building technical software for your users, you need to know what their level of expertise and experience is. If you can use a job title to describe their position, that is a great start.
You should define the users’ strengths and weaknesses, that is, what you expect them to excel at and where they will require more of your assistance to succeed at their task.
Define the scenarios in which your users use the software you will build. What information do they have available? Which information do they need to gather? What kind of decisions do they need to make? Which decisions can be automated for them? Do they need to run the software in a specific environment? Do they need access to data to do their task? How is this data made available to them? When do they use the software? How frequently do they use the software? How much of their time is spent on the task they are solving using the software you’re building?
Define what the users’ goals are. You might be creating a text editor, if your users’ goal is to transmit information between companies, they may not care at all about making the text fancy but they might appreciate your editor helping them correct grammatical mistakes.
You want to be as specific as possible when describing your users so that your decisions are guided by this persona you’re creating. What kind of company are they working in? Startups, PME, large enterprises? Do they have multiple responsibilities or are specialized?
With all this information about your users you should be able to make more judicious decisions. This will help you scope your work.
References
#
Problem
#
I have rated a lot of movies on Rotten Tomatoes and I’d like to export them to a text format. I also know PHP. How can I do that?
Solution
#
In 2013 I wrote Rotten Tomatoes ratings exporter for the purpose of exporting the movies I had rated into a format I could read and also store.
To automate the process of acquiring the ratings, I wrote a small class which needs information available in the site’s cookies once you are logged in. With this information in hand, it is then possible to fetch the ratings from the website and store them in any desired format, given that the data returned to you is a plain PHP array.
References
#