Oh, yeah, I still have this blog thing.
As is fairly obvious to people who've seen my homepage, I take quite a lot of photographs. While I'm a big fan of using file system hierachies to categorise information, and it's fundamental to the way my Gallery software works, there are nuances that are hard to represent with a single hierachy.
For years, I've used f-spot to maintain a tagged database of my photos, which makes various queries a lot simplere. Of course, f-spot has always been annoyingly buggy, and is now essentially dead and increasingly suffering from bit-rot. The official replacement, shotwell, is nice enough, unless you're the sort of person you cares about how the data is actually stored in the database, in which case it's not for you. (I shall not eloborate - the curious can easily shotwell's behaviour themselves). Since I'm the sort of that does care, shotwell's out. There is digikam, but Ive never hit it off with the KDE design philosophy, so that doesn't work well for me either.
Which of course leaves the traditional solution to minor inconviences with other people's software. Re-inventing the wheel. This does have a number of advantages. I can have something that just has the features I want, and drop a lot of stuff, such as all the image editing functionality, that there are better tools for. Furthermore, by writing in python 3, using gobject introspection and a pre-release of SQLObject, I can justify it both as a learning exercise (since I hadn't previously gotten beyond toy gtk3 programs), and also as testing SQLObject 3.
The net result of all this is Phototagger , something I'd actually decided to write in 2014 (but then as a python 2 program), but then didn't do any work on for more than a year, and which now represents far too much of my free time over the last month and a half or so.
There was a point early on where the actual code design was reasonably neat, but stuff got tangled as I worked around some questionable early choices without addressing the underlying problem, so a rewrite is on the cards at some point, but it now works pretty nicely.
The features are nothing special - photos can be added, tagged, tags can be edited and have various images assigned to them, and all that sort of thing. The search functionality is a bit limited - it doesn't allow constructing arbitary tag queries, but it covers the sort of queries I generally need.
Indivual photos can be viewed
And zoomed - this also supports mouse panning, which took a surprisingly long time to get right due to a misreading of the Gtk documentation.
As a learning experience, it was pretty useful. I got to play with some of the new Gtk 3 widgets, such as the Stack and FlowBox widgets, which are nifty, and work through some of the changes between Gtk 2 and Gtk 3. It's also given me a fair amount of confidence in SQLObject's python 3 support, although it doesn't stress the limits of SQLObject particularly.
As a tagging tool, it's now functional enough to replace f-spot for me, and has proved a great deal more stable. I now just need to rewrite a bunch of the tools I wrote to managing my f-spot database easier, and then I can finally move on.
Showing posts with label python. Show all posts
Showing posts with label python. Show all posts
Thursday, January 28, 2016
Sunday, April 21, 2013
Pyweek 16: Nemesis
So we've reached the end of another pyweek, and I've managed another game ("Bane's Befuddlement"), so it's time for my usual post-pyweek post.
Since most of the usual suspects were busy with other things, I ended up doing a solo entry for the first time. I also decided, on something of a poorly thought-out whim, to use kivy for this entry.
Based on what happened with Mad Science (pyweek 14), I expected the solo entry to be something of struggle, and it does show in the end result. Artwork was obviously limited, since it's not my strong suite and I didn't want to spend a lot of time on the issue, but fortunately I was able to offset that somewhat by using one of the existing dungeon tile collections.
Probably the most crucial failing is ambition. Several of the past pyweek entries I'm been involved in have suffered from being overly ambitious (Nine Tales (pyweek 12) being the worst offender), so I aimed to avoid that pitfall this time. As a result, I was under-ambitious. The game isn't that challenging, and is a bit bland. It's not without potential, but needed more time spent on the initial concept to really get something I would have been happy with.
There are several things I am happy with. I reaffirmed my belief in the importance of level editors for pyweek games. Even with the very simplistic one I wrote here, I was able to fill out the level content very quickly on Friday and Saturday as a result, and I rather like the careful level of stupid that AI achieves.
There are a few tricks about touch interfaces I need to learn. The most obvious one from this game is to avoid anything that goes too close to the edge of the screen - it's too easy to accidentally trigger one of the other buttons in that case.
As an exercise in working more with kivy, this was pretty successful. I'm still uncertain about how I feel about kivy. It's quite nice in some ways - being able to add a settings pane or a popup dialog with only a couple of lines of code - but it's also all sorts of magic in ways that really aren't helpful. The complicated import dance needed to ensure that we set the configured window size before creating the kivy window is just one example of this sort of thing. More annoyingly, the inability to hook in to the command-line argument parsing without some serious monkey patching is the sort of poorly thought out design that makes working with it painful.
Python's still a work in progress on android, and, while it's possible to use pyjnius to access the hardware, and to build apk's, both require more fiddling with building python for android than I was prepared to do during a pyweek, so, while the game runs OK on my phone, it's missing some features which would be very nice to have. There is enough work happening in this space that this should improve, though.
Windows packaging with kivy is a pain, and the recommended approach of using pyinstaller just failed horribly for me. I'm not sure if where to lay the blame for that, but debugging windows issues is not my favourite activity, and anyway not something I'm capable of doing at the end of a pyweek.
Since most of the usual suspects were busy with other things, I ended up doing a solo entry for the first time. I also decided, on something of a poorly thought-out whim, to use kivy for this entry.
Based on what happened with Mad Science (pyweek 14), I expected the solo entry to be something of struggle, and it does show in the end result. Artwork was obviously limited, since it's not my strong suite and I didn't want to spend a lot of time on the issue, but fortunately I was able to offset that somewhat by using one of the existing dungeon tile collections.
Probably the most crucial failing is ambition. Several of the past pyweek entries I'm been involved in have suffered from being overly ambitious (Nine Tales (pyweek 12) being the worst offender), so I aimed to avoid that pitfall this time. As a result, I was under-ambitious. The game isn't that challenging, and is a bit bland. It's not without potential, but needed more time spent on the initial concept to really get something I would have been happy with.
There are several things I am happy with. I reaffirmed my belief in the importance of level editors for pyweek games. Even with the very simplistic one I wrote here, I was able to fill out the level content very quickly on Friday and Saturday as a result, and I rather like the careful level of stupid that AI achieves.
There are a few tricks about touch interfaces I need to learn. The most obvious one from this game is to avoid anything that goes too close to the edge of the screen - it's too easy to accidentally trigger one of the other buttons in that case.
As an exercise in working more with kivy, this was pretty successful. I'm still uncertain about how I feel about kivy. It's quite nice in some ways - being able to add a settings pane or a popup dialog with only a couple of lines of code - but it's also all sorts of magic in ways that really aren't helpful. The complicated import dance needed to ensure that we set the configured window size before creating the kivy window is just one example of this sort of thing. More annoyingly, the inability to hook in to the command-line argument parsing without some serious monkey patching is the sort of poorly thought out design that makes working with it painful.
Python's still a work in progress on android, and, while it's possible to use pyjnius to access the hardware, and to build apk's, both require more fiddling with building python for android than I was prepared to do during a pyweek, so, while the game runs OK on my phone, it's missing some features which would be very nice to have. There is enough work happening in this space that this should improve, though.
Windows packaging with kivy is a pain, and the recommended approach of using pyinstaller just failed horribly for me. I'm not sure if where to lay the blame for that, but debugging windows issues is not my favourite activity, and anyway not something I'm capable of doing at the end of a pyweek.
Sunday, May 13, 2012
Pyweek 14: Mad Science
So we've reached the end of another pyweek, and managed yet another game ("Tomorrow, the World!!!"), so it's time for my usual post-pyweek post.
It's not one of the better entires I've been involved in, and I expect us to place very middle of the field this year. I think there's a decent concept here, but we didn't get enough content or game balance to have something that's really fun, and, as a result, much of the early game is a bit random and frustrating. There's nothing unfixable, but it needs a couple of days of people playing and extending the game to get to the point I'd actually be happy with it. It's got its entertaining bits, though, so I'm happy enough to throw it out there to be judged and see what happens.
Due to everyone being busy and / or being out of the country, we had a much smaller team than previous entries. I expected we would need to downscale as a result, but this proved to be more of a problem than I had actually anticipated. There just wasn't space for people to play the game much, which partly accounts for the issues, and there also much less opportunity to riff off other people's ideas to improve things. Everyone being busy in the day didn't help either. There were far fewer of the "ooh, a bunch of cool stuff has happened since I last worked on the game" moments, which also made it feel like more of a slog than past entries. The lack of people was most felt in the final Saturday finishing sprint, for which only 2 of us were available - most conversations were "I'm fixing X, can you look at Y?", and then silence for about 30 minutes while stuff happened. Given how the smaller team affected things, my respect for people doing successful solo entries has gone up even further.
I've now been involved 5 pyweek entries, and they've all been content heavy in various ways, so it's fair to call that as a definite trend. Given the games I enjoy playing, and the high value I place on being amusing, that's not surprising, but it's not a good feature for games with small teams.
On the upside, it was still fun. Coding for pyweek is always liberating, since it's much more important to have something close now than completely correct later. Having now done 3 pyweeks with mercurial, we've now also got the setup and workflow around that reasonably well sorted (which means we're almost certainly due to change to something else now), and past experience means that we're making a reasonable number correct design decisions early, so I'm quite happy with the final structure of the game.
It's not one of the better entires I've been involved in, and I expect us to place very middle of the field this year. I think there's a decent concept here, but we didn't get enough content or game balance to have something that's really fun, and, as a result, much of the early game is a bit random and frustrating. There's nothing unfixable, but it needs a couple of days of people playing and extending the game to get to the point I'd actually be happy with it. It's got its entertaining bits, though, so I'm happy enough to throw it out there to be judged and see what happens.
Due to everyone being busy and / or being out of the country, we had a much smaller team than previous entries. I expected we would need to downscale as a result, but this proved to be more of a problem than I had actually anticipated. There just wasn't space for people to play the game much, which partly accounts for the issues, and there also much less opportunity to riff off other people's ideas to improve things. Everyone being busy in the day didn't help either. There were far fewer of the "ooh, a bunch of cool stuff has happened since I last worked on the game" moments, which also made it feel like more of a slog than past entries. The lack of people was most felt in the final Saturday finishing sprint, for which only 2 of us were available - most conversations were "I'm fixing X, can you look at Y?", and then silence for about 30 minutes while stuff happened. Given how the smaller team affected things, my respect for people doing successful solo entries has gone up even further.
I've now been involved 5 pyweek entries, and they've all been content heavy in various ways, so it's fair to call that as a definite trend. Given the games I enjoy playing, and the high value I place on being amusing, that's not surprising, but it's not a good feature for games with small teams.
On the upside, it was still fun. Coding for pyweek is always liberating, since it's much more important to have something close now than completely correct later. Having now done 3 pyweeks with mercurial, we've now also got the setup and workflow around that reasonably well sorted (which means we're almost certainly due to change to something else now), and past experience means that we're making a reasonable number correct design decisions early, so I'm quite happy with the final structure of the game.
Sunday, April 10, 2011
Pyweek 12: Nine Times
So a bunch of us took part in pyweek 12. While we had a lot of fun, and the ended up with something that looked very good, we didn't quite manage to get the game completely playable inside the 1 week time limit.
Failures are generally more educational than successes, so some thoughts on what went wrong.
a) We were arguably over ambitious, although I'm not sure where we needed to scale down. This is probably the most important factor as it meant we took too long to get enough of the game playable that we could start getting stuff hooked up, and it also meant that we didn't have as much overlap on knowledge of the code base as we had on Suspended Sentence, which slowed us down at critical points, and made fixing various bugs quite hard.
b) We took too long to converge on what game we were writing. Various bits grew in different directions and got fixed in the wrong place, and there were a couple of dead ends we pursued. We also didn't have the time to refactor code to fix things as we've had previously, so the game is more than just somewhat buggy.
c) We didn't know mercurial nearly as well as we thought we did. Most of us had used it for the Genshi sprint, which was quite intense, as well as using it on other projects, so we thought we going for a safe option At the pace of pyweek development, however, we tripped over mercurial not behaving as we expected quite often, and the slow downs fixing the issues this caused all added up to quite a lot of lost time.
Still, the game has a number of promising bits, and, with some concerted work on the engine, and some work on the other bugs and the game balance, it can turn into something really quite cool. This should be a good candidate for pyggy, if we can summon the energy to tackle again.
Failures are generally more educational than successes, so some thoughts on what went wrong.
a) We were arguably over ambitious, although I'm not sure where we needed to scale down. This is probably the most important factor as it meant we took too long to get enough of the game playable that we could start getting stuff hooked up, and it also meant that we didn't have as much overlap on knowledge of the code base as we had on Suspended Sentence, which slowed us down at critical points, and made fixing various bugs quite hard.
b) We took too long to converge on what game we were writing. Various bits grew in different directions and got fixed in the wrong place, and there were a couple of dead ends we pursued. We also didn't have the time to refactor code to fix things as we've had previously, so the game is more than just somewhat buggy.
c) We didn't know mercurial nearly as well as we thought we did. Most of us had used it for the Genshi sprint, which was quite intense, as well as using it on other projects, so we thought we going for a safe option At the pace of pyweek development, however, we tripped over mercurial not behaving as we expected quite often, and the slow downs fixing the issues this caused all added up to quite a lot of lost time.
Still, the game has a number of promising bits, and, with some concerted work on the engine, and some work on the other bugs and the game balance, it can turn into something really quite cool. This should be a good candidate for pyggy, if we can summon the energy to tackle again.
Sunday, August 29, 2010
Pyweek 11: Caught
After the success of last year's entry, we decided to do another one this year (we skipped pyweek 10, as none of us had time available then), with a slightly tweaked team.
This years entry ran rather differently. We met on Sunday to discuss things, and,. although we settled on a point and click adventure game fairly early, we spent a lot of time converging to a basic premise and initial plot, although we did manage to get this largely settled by Sunday evening.
Another serious issue was the colds several members of the team (including me) were battling with. This added a layer of fuzzy thinking in several places, and was just generally not helpful.
The development ran rather differently to the last pyweek. I think because of the previous experience, the pace of development, while equally fast, wasn't nearly as startling. We also used a different toolkit, albow, which, while not perfect, was easier to get on top of than pgu had been. The game, being point and click, is also less busy than fox assault was, so from that point of view the code base was easier to manage. As a result, the basic engine part of the game actually shaped up fairly quickly, and we were generally able to extend it as needed with little trouble.
On the other hand, as an adventure game, we needed to generate a lot of content before we could hook up parts of the game. This was the largest stumbling block for the game. By Thursday, it looked like we would have something workable, but we only got rid of the last of the text placeholders quite late on Saturday. This meant we didn't have a reasonably playable version of the game until quite late, and thus never got much feedback as a result of other people playing the game. This helped contribute to the annoying bugs in the final submission, as, by that stage, we weren't testing unexpected interactions enough.
Notes on things that happened along the way:
While not as fun for me as the first pyweek, largely due to the cold, this was still enjoyable, and the game, while quite a simple little adventure game, is amusing, and has promise as a more general adventure game engine.
This years entry ran rather differently. We met on Sunday to discuss things, and,. although we settled on a point and click adventure game fairly early, we spent a lot of time converging to a basic premise and initial plot, although we did manage to get this largely settled by Sunday evening.
Another serious issue was the colds several members of the team (including me) were battling with. This added a layer of fuzzy thinking in several places, and was just generally not helpful.
The development ran rather differently to the last pyweek. I think because of the previous experience, the pace of development, while equally fast, wasn't nearly as startling. We also used a different toolkit, albow, which, while not perfect, was easier to get on top of than pgu had been. The game, being point and click, is also less busy than fox assault was, so from that point of view the code base was easier to manage. As a result, the basic engine part of the game actually shaped up fairly quickly, and we were generally able to extend it as needed with little trouble.
On the other hand, as an adventure game, we needed to generate a lot of content before we could hook up parts of the game. This was the largest stumbling block for the game. By Thursday, it looked like we would have something workable, but we only got rid of the last of the text placeholders quite late on Saturday. This meant we didn't have a reasonably playable version of the game until quite late, and thus never got much feedback as a result of other people playing the game. This helped contribute to the annoying bugs in the final submission, as, by that stage, we weren't testing unexpected interactions enough.
Notes on things that happened along the way:
- My quick 20 minute hack to help layout rectangles grew rather more features than I expected.
- Finding sounds with decent licenses is a painful and tedious exercise.
- The same is true for fonts
- But writing a little sine wave tone generator using just sys, math and struct was amusing
- Hooking up humorous descriptions and interactions is why to kill compiling time
While not as fun for me as the first pyweek, largely due to the cold, this was still enjoyable, and the game, while quite a simple little adventure game, is amusing, and has promise as a more general adventure game engine.
Sunday, September 6, 2009
Pyweek 9: Feather
Pyweek has, since I first heard about it, been one of things I've always wanted to have a stab at, but never got around to. For pweek 9, with the benefit of enthusiasm from fellow members of CTPUG, we managed to get an entry together. We ended up with a rather fun little strategy game (in my completely neutral and unbiased opinion, of course), called "Fox Assault".
The whole development experience was quite interesting - I'm still somewhat amazed on how quickly we seemed to hit a concept in the initial discussion and were able to just run with it. Based on this experience, and the momentum the initial concept discussions gave us, for any future team entries, having an initial in person discussion & planning session is will be most useful.
The pace of development was more than a bit breathless, and, on a couple of occasions, I watched a throw-away comment in the code suddenly arrive in the codebase turned into something awesome, which was quite gratifying.
The way the game took over my week was a little unexpected, and contributed to rather less sleep than was probably healthy (the VTES social and a C. J. Cherryh novel didn't help this either). Some of other events of the week really quite surprised me, such as #ctpug suddenly becoming our personal source of additional testers.
Our one major error was not looking taking more time to familiarise ourselves with pygame and the available support libraries more beforehand. We chose to use Phil's pyGame Utilities almost on a whim, and, while it helped a great deal in providing several useful framework tools, and did save us quite a lot of time initially, it also forced several compromises and hacks onto us, partly due to design decisions taken by pgu, and partly due the time pressure favouring "It works, that's good enough" over pretty much anything else.
Overall, the whole experience was simply gosh-darn FUN, and while I would like to see our game score well, I'm overall really not bothered about that, since I know it's quite cool. I'd certainly like to do this again, although I'll probably try and plan to have a day's leave towards the end of the week for the next one.
The whole development experience was quite interesting - I'm still somewhat amazed on how quickly we seemed to hit a concept in the initial discussion and were able to just run with it. Based on this experience, and the momentum the initial concept discussions gave us, for any future team entries, having an initial in person discussion & planning session is will be most useful.
The pace of development was more than a bit breathless, and, on a couple of occasions, I watched a throw-away comment in the code suddenly arrive in the codebase turned into something awesome, which was quite gratifying.
The way the game took over my week was a little unexpected, and contributed to rather less sleep than was probably healthy (the VTES social and a C. J. Cherryh novel didn't help this either). Some of other events of the week really quite surprised me, such as #ctpug suddenly becoming our personal source of additional testers.
Our one major error was not looking taking more time to familiarise ourselves with pygame and the available support libraries more beforehand. We chose to use Phil's pyGame Utilities almost on a whim, and, while it helped a great deal in providing several useful framework tools, and did save us quite a lot of time initially, it also forced several compromises and hacks onto us, partly due to design decisions taken by pgu, and partly due the time pressure favouring "It works, that's good enough" over pretty much anything else.
Overall, the whole experience was simply gosh-darn FUN, and while I would like to see our game score well, I'm overall really not bothered about that, since I know it's quite cool. I'd certainly like to do this again, although I'll probably try and plan to have a day's leave towards the end of the week for the next one.
Monday, December 22, 2008
What I've done on my Xmas Holiday
As has been mentioned several times, I tend to spend a lot of time hacking on Sutekh, the VtES card manager.
I am, however, also somewhat addicted to the (now defunct) Dune CCG, so some time ago, I vaguely started working on a fork of Sutekh to deal with Dune's cards.
And, with minimal cheating [1], I now present the first evidence of Thufir's evolution to non-vapourware status [2]:

[1] It doesn't parse unmodified html files yet - I'm leaning towards just distributing the tweaked files as a better choice fixing the parser to handle the rather horrible html.
[2] git repo
I am, however, also somewhat addicted to the (now defunct) Dune CCG, so some time ago, I vaguely started working on a fork of Sutekh to deal with Dune's cards.
And, with minimal cheating [1], I now present the first evidence of Thufir's evolution to non-vapourware status [2]:

[1] It doesn't parse unmodified html files yet - I'm leaning towards just distributing the tweaked files as a better choice fixing the parser to handle the rather horrible html.
[2] git repo
Sunday, November 16, 2008
Development style
A couple of weeks ago, I decided to make a concerted effort to fill in some of Sutekh's documentation, since we are planning to release fairly soon.
As a result, Sutekh now displays icons in the main tree view, and has a plugin for importing card sets from zip files.
I'm prone to this development pattern, and, given the amount of feature creep you see in modern software, I suspect that it's actually quite common.
(There has also been progress on the documentation, though, and the icons feature is really nifty)
As a result, Sutekh now displays icons in the main tree view, and has a plugin for importing card sets from zip files.
I'm prone to this development pattern, and, given the amount of feature creep you see in modern software, I suspect that it's actually quite common.
(There has also been progress on the documentation, though, and the icons feature is really nifty)
Thursday, October 16, 2008
Reasons for loving FLOSS, #<fairly large number>
As I've mentioned before, I've been poking at a gmail backup application based off libgmail. It's been making slow progress, partly because I've been distracted by other things (such as Project Euler), but it hasn't completely stalled.
Recently, while fiddling with it, though, I hit a snag - libgmail would throw an exception when trying to access certain threads. After much head scratching, and testing, I eventually twigged that the problematic threads all included messages sent via google chat. A little focussed debugging showed that gmail creates pseudo mail messages for these when accessed via the API libgmail uses that don't quite match the messages libgmail expects. A little crude patch later, and the problem was fixed for me.
Thus armed, I submitted a bug to the Debian bug tracker, at around 19:00 local time. At 21:00, I received a acknowledgement from the maintainer, who had forwarded upstream, and at 21:10, I received a note that a patch had been applied upstream. Just over 2 hours from bug report to upstream fix strikes me as pretty good going.
Recently, while fiddling with it, though, I hit a snag - libgmail would throw an exception when trying to access certain threads. After much head scratching, and testing, I eventually twigged that the problematic threads all included messages sent via google chat. A little focussed debugging showed that gmail creates pseudo mail messages for these when accessed via the API libgmail uses that don't quite match the messages libgmail expects. A little crude patch later, and the problem was fixed for me.
Thus armed, I submitted a bug to the Debian bug tracker, at around 19:00 local time. At 21:00, I received a acknowledgement from the maintainer, who had forwarded upstream, and at 21:10, I received a note that a patch had been applied upstream. Just over 2 hours from bug report to upstream fix strikes me as pretty good going.
Saturday, October 11, 2008
Adding periodicity
Still generating complete sequences, so linear growth is expected.
There's a bunch of improvements that can be made to the code - especially the periodicity check is not that efficient, and tends to repeat work which could be avoided. This bites for the U(2, 21) sequence, for which the direct approach presented earlier can calculate 20000000 elements in around 290 seconds, but the overall problem is now something that can be solved in "reasonable" time periods
Presented without further comment.
Found period 32, difference 126
time to generate 200000 numbers (U (2, 5)) is 0.37129
time to generate 2000000 numbers (U (2, 5)) is 3.66487
time to generate 20000000 numbers (U (2, 5)) is 38.38641
Found period 26, difference 126
time to generate 200000 numbers (U (2, 7)) is 0.36446
time to generate 2000000 numbers (U (2, 7)) is 4.06793
time to generate 20000000 numbers (U (2, 7)) is 44.24620
Found period 444, difference 1778
time to generate 200000 numbers (U (2, 9)) is 0.40023
time to generate 2000000 numbers (U (2, 9)) is 5.73586
time to generate 20000000 numbers (U (2, 9)) is 38.33744
Found period 1628, difference 6510
time to generate 200000 numbers (U (2, 11)) is 0.41977
time to generate 2000000 numbers (U (2, 11)) is 3.65161
time to generate 20000000 numbers (U (2, 11)) is 37.85133
Found period 5906, difference 23622
time to generate 200000 numbers (U (2, 13)) is 0.65191
time to generate 2000000 numbers (U (2, 13)) is 3.69802
time to generate 20000000 numbers (U (2, 13)) is 36.82868
Found period 80, difference 510
time to generate 200000 numbers (U (2, 15)) is 0.36584
time to generate 2000000 numbers (U (2, 15)) is 3.61351
time to generate 20000000 numbers (U (2, 15)) is 37.08529
time to generate 200000 numbers (U (2, 17)) is 4.52874
Found period 126960, difference 507842
time to generate 2000000 numbers (U (2, 17)) is 7.54118
time to generate 20000000 numbers (U (2, 17)) is 39.30235
time to generate 200000 numbers (U (2, 19)) is 4.54097
Found period 380882, difference 1523526
time to generate 2000000 numbers (U (2, 19)) is 35.07058
time to generate 20000000 numbers (U (2, 19)) is 36.92766
time to generate 200000 numbers (U (2, 21)) is 4.10050
time to generate 2000000 numbers (U (2, 21)) is 43.80197
Found period 2097152, difference 8388606
time to generate 20000000 numbers (U (2, 21)) is 456.09959
There's a bunch of improvements that can be made to the code - especially the periodicity check is not that efficient, and tends to repeat work which could be avoided. This bites for the U(2, 21) sequence, for which the direct approach presented earlier can calculate 20000000 elements in around 290 seconds, but the overall problem is now something that can be solved in "reasonable" time periods
Presented without further comment.
# time python ./solve_167.py
Answer = <answer redacted, but confirmed correct>
real 7m15.052s
user 6m41.213s
sys 0m15.365s
Friday, October 10, 2008
This may actually be feasible now
With CTPUG 14 in the near future, and two failed attempts to tackle the Ulam seuqence problem from the previous meetings, I decided to give the problem some thought today.
So, from today's bright idea,:
(Sequences U(2,5) and U(2,7) verified against the On-Line Encyclopedia of Integer Sequences (to 1000 elements))
The period detection and related footwork is still missing, but with a linear time growth, the problem looks much more doable.
So, the bright idea:
Looking at various articles, it's mentioned several times that Scherml and Speigel proved that "The sequence (2, v) for odd v >= 5 had precisely 2 even terms".
It can also be shown that the second even number occurs fairly early in the sequence.
Since we're dealing with sequences of that form, after the initial terms to reach the second even term, the candidate numbers we're dealing with must be odd. Generating the new candidates is thus trivial, and, with a little housekeeping, the list of candidates to consider can be kept short on every loop.
I'm not sure whether to feel chuffed about spotting this, or annoyed that I didn't spot this earlier.
So, from today's bright idea,:
$ python ./Ulam.py
time to generate 100 numbers (U (2, 5)) is 0.00426
time to generate 1000 numbers (U (2, 5)) is 0.03167
time to generate 10000 numbers (U (2, 5)) is 0.22796
time to generate 100000 numbers (U (2, 5)) is 1.44548
time to generate 1000000 numbers (U (2, 5)) is 14.71493
time to generate 100 numbers (U (2, 7)) is 0.00214
time to generate 1000 numbers (U (2, 7)) is 0.01558
time to generate 10000 numbers (U (2, 7)) is 0.15068
time to generate 100000 numbers (U (2, 7)) is 1.43910
time to generate 1000000 numbers (U (2, 7)) is 14.62419
time to generate 100 numbers (U (2, 9)) is 0.00245
time to generate 1000 numbers (U (2, 9)) is 0.01770
time to generate 10000 numbers (U (2, 9)) is 0.15190
time to generate 100000 numbers (U (2, 9)) is 1.45868
time to generate 1000000 numbers (U (2, 9)) is 14.93112
time to generate 100 numbers (U (2, 11)) is 0.00211
time to generate 1000 numbers (U (2, 11)) is 0.01602
time to generate 10000 numbers (U (2, 11)) is 0.15409
time to generate 100000 numbers (U (2, 11)) is 1.49055
time to generate 1000000 numbers (U (2, 11)) is 15.08445
...
time to generate 100 numbers (U (2, 21)) is 0.00486
time to generate 1000 numbers (U (2, 21)) is 0.03741
time to generate 10000 numbers (U (2, 21)) is 0.25073
time to generate 100000 numbers (U (2, 21)) is 1.78683
time to generate 1000000 numbers (U (2, 21)) is 17.9669
(Sequences U(2,5) and U(2,7) verified against the On-Line Encyclopedia of Integer Sequences (to 1000 elements))
The period detection and related footwork is still missing, but with a linear time growth, the problem looks much more doable.
So, the bright idea:
Looking at various articles, it's mentioned several times that Scherml and Speigel proved that "The sequence (2, v) for odd v >= 5 had precisely 2 even terms".
It can also be shown that the second even number occurs fairly early in the sequence.
Since we're dealing with sequences of that form, after the initial terms to reach the second even term, the candidate numbers we're dealing with must be odd. Generating the new candidates is thus trivial, and, with a little housekeeping, the list of candidates to consider can be kept short on every loop.
I'm not sure whether to feel chuffed about spotting this, or annoyed that I didn't spot this earlier.
Wednesday, October 8, 2008
Paging Captain "Ya know, that's really pretty obvious"
Occasionally, it's useful to be able to write a one-line script (it's especially handy for pasting into IRC channels). Python's one-line capabilities are somewhat limited though. Statement lists (see the offical grammer) are always terminated by a newline. Thus, it's impossible to directly express the following snippet as a single line:
Of course, there are a vast number of ways to achieve the same effect using the constructs that can be placed on a single line, and, other than showing off, there's usually very little need to coerce something to be a single line.
Today, however, it occurred to me that it's also possible to abuse eval for this (I don't know why this only struck me today), leading to:
While not exactly readable, I think combination of eval, compile + a one-liner is just too amusing to ignore. I'm sure that eventually, somewhere, I'll be able to convince myself that this trick will be just what I need.
z=0
for x in range(5):
z += x
print z
Of course, there are a vast number of ways to achieve the same effect using the constructs that can be placed on a single line, and, other than showing off, there's usually very little need to coerce something to be a single line.
Today, however, it occurred to me that it's also possible to abuse eval for this (I don't know why this only struck me today), leading to:
python -c 'eval(compile("z=0\nfor x in range(5):\n z+=x\nprint z", "test", "exec"))'
While not exactly readable, I think combination of eval, compile + a one-liner is just too amusing to ignore. I'm sure that eventually, somewhere, I'll be able to convince myself that this trick will be just what I need.
Saturday, September 13, 2008
The Side project, or a tale of paranoia and python
This starts with a post to clug-tech (http://lists.clug.org.za/pipermail/clug-tech/2008-September/041635.html), titled "Make automatic backups of gmail account". This extracted several obvious solutions using getmail or fetchmail & gmail's IMAP service. There are obvious weaknesses with using IMAP to backup mail, though - exposing all the labels as folders is not desirable when you regularly use multiple labels per message, and any re-arrangement of mail into per month mailboxes or the like needs to be handled client side, which seems a waste given that gmail has a rather nice search engine as a major selling point.
Combine this with the following ingredients:
Combine this with the following ingredients:
- I'm prone to sporadic bursts of data paranoia
- I've wanted to play with libgmail (a python interface to gmail) for ages
- I don't have a backup of my gmail account's mail
- work has involved lots of meetings & academic proposal writing, and not much of actual interest of late.
Saturday, June 21, 2008
Python Sprint Day Take 2
In terms of overall productivity, this was less successful than the first. Partly the break for the talks ate up a lot of time, so getting momentum was a lot harder, and also, with the recent beta release, the number of clear issues was much reduced. The checkin that broke floatobject for some of us didn't help either, as people got side tracked into getting working python 2.6 builds.
Still, we prodded a couple of issues from the last bug day, and a couple of new bugs were filed about odd bits in the code (mainly by Simon), so there was some positive progress.
The CTPUG meeting part of the day was a definite success. The localisation talk took some time to get going, and it would have been nice if Dwayne had been able to stay longer so that we could have explored some of the discussion avenues a bit further. Simon's PyObject talk was interesting, especially touching on some of the differences between py3k and the python 2.X series.
Still, we prodded a couple of issues from the last bug day, and a couple of new bugs were filed about odd bits in the code (mainly by Simon), so there was some positive progress.
The CTPUG meeting part of the day was a definite success. The localisation talk took some time to get going, and it would have been nice if Dwayne had been able to stay longer so that we could have explored some of the discussion avenues a bit further. Simon's PyObject talk was interesting, especially touching on some of the differences between py3k and the python 2.X series.
Sunday, May 11, 2008
Redoing my photo gallery
I have, for some years now, maintained an online gallery of my photographs on dip.
It has, for most of it's existence, been maintained by a rather complex and fragile set of shell scripts, so , with the recent spate of long weekends, I finally got around to replacing them with some more maintainable python code. This, after poking at the problem off and on for the last two weeks, reached a point that I started moving my gallery over to this on Friday. While the code is not bug-free yet [1], it works pretty well.
Why write yet another gallery generator, though?
[1] For instance, I had to fix a memory issue this evening. Keeping references to 100 odd 2592x1944 images after PIL's loaded them completely is not the brightest thing I've ever done.
It has, for most of it's existence, been maintained by a rather complex and fragile set of shell scripts, so , with the recent spate of long weekends, I finally got around to replacing them with some more maintainable python code. This, after poking at the problem off and on for the last two weeks, reached a point that I started moving my gallery over to this on Friday. While the code is not bug-free yet [1], it works pretty well.
Why write yet another gallery generator, though?
- Why not? With python's tools, most of the heavy lifting can be passed off to PIL and elementtree, and it's just the file parsing that needs attention.
- Most existing packages are trying to solve a different problem, namely managing the gallery entirely through a web browser. I don't need to do that, and so focus on creating static pages offline.
- I'd have to write a tool to import the existing image annotations into whatever gallery I use.
- I've wanted to experiment with git for a while, so this project also gave me an opportunity to poke around at that.
[1] For instance, I had to fix a memory issue this evening. Keeping references to 100 odd 2592x1944 images after PIL's loaded them completely is not the brightest thing I've ever done.
CTPUG's python sprint day
Overall, I think the day went well. At one point we had 8 people in the room working on various issues, and, during the course of the day, we touched 11 issues by my count (http://groups.google.com/group/ctpug/browse_thread/thread/edff4863226806ea), and submitted patches (in some cases multiple patches) to 10 of them.
Due to the time-zone issues, there wasn't much opportunity to push for review of the patches, but they are now there.
Due to the time-zone issues, there wasn't much opportunity to push for review of the patches, but they are now there.
Tuesday, March 25, 2008
Misc Geekery
(or wasting time with elementtree)
Like many people, I have attempted and pretty much utterly failed to maintain the classic list of links style web-page. This is most notable example is my list of web-comics I read, which is both out of date, and horribly incomplete.
Part of this, is, of course, the effort to manually create the webpages needed, and fill in the content. However, since galeon's bookmark are stored in nicely structured xml, and I group the comics I follow in a not completely broken manner, I hit on the bright idea of pulling the bookmarks out using elementtree, and, furthermore, I can use galeon's bookmark notes to do the documenting about the comics.
So, after a couple of hours of prodding (much of it spent trying to get an overly complex design working before an important light-bulb went on), I now have a little python script that pulls all the data out, and writes a nice, standards conforming html file with all details - Now I just need to actually go through and sort out my comics bookmarks a bit, and I'll actually have a much more maintainable way of generating the list.
Of course, when I eventually move away from galeon (I've been using epiphany exclusively on my laptop for a while now, so it's likely to be not too far off), I'll have to tweak the script, and probably fiddle with the bookmark's conversion a bit to get all the comments across properly, but such is progress.
ah, the idle amusements of a life around computers.
(and to think I'd planned to spent the evening working on Sutekh)
Like many people, I have attempted and pretty much utterly failed to maintain the classic list of links style web-page. This is most notable example is my list of web-comics I read, which is both out of date, and horribly incomplete.
Part of this, is, of course, the effort to manually create the webpages needed, and fill in the content. However, since galeon's bookmark are stored in nicely structured xml, and I group the comics I follow in a not completely broken manner, I hit on the bright idea of pulling the bookmarks out using elementtree, and, furthermore, I can use galeon's bookmark notes to do the documenting about the comics.
So, after a couple of hours of prodding (much of it spent trying to get an overly complex design working before an important light-bulb went on), I now have a little python script that pulls all the data out, and writes a nice, standards conforming html file with all details - Now I just need to actually go through and sort out my comics bookmarks a bit, and I'll actually have a much more maintainable way of generating the list.
Of course, when I eventually move away from galeon (I've been using epiphany exclusively on my laptop for a while now, so it's likely to be not too far off), I'll have to tweak the script, and probably fiddle with the bookmark's conversion a bit to get all the comments across properly, but such is progress.
ah, the idle amusements of a life around computers.
(and to think I'd planned to spent the evening working on Sutekh)
Friday, November 23, 2007
Sutekh ramblings
(Sutekh is, of course, the python Vampire card management app Simon Cross and I have been working on)
Conceptually, the app is fairly simple, and I've been largely using it as a learning bed for pygtk. Consequently, it does things by hand, rather than the often recommended "use glade" approach. Personally, for something that now as a number of dynamic aspects to the gui (ability to rearrange panes, and so on), I think glade would have become quite painful, and straight pygtk coding is fairly easy once you wrap your head around the whole container paradigm. As an vehicle for learning pygtk, it has been pretty good.
But what has surprised me is it's ability to uncover database oddities. We have to date, discovered two issues with sqlobject (both fixed), had to work around sqlite's query optimiser by throwing in extra joins, triggered a segfault in sqlite (fixed in more recent sqlite's fortunately) and hit a bug in mysql query optimiser that makes certain queries run extremely slowly. WE ahven't triggered any postgresql bugs yet (although there have been a few weirdness's with certain versions of sqlobject talking to postgresql), but have had to fix several bugs in Sutekh's code because of postgres's much more stringent SQL implementation. For not a teribly complex database app, it's quite a impressive list.
So, do other people not trigger these bugs? Are we stretching corners of the databases that many other apps just don't hit (Sutekh has a number of tables, and can involve some quite complex joins)? Do other people just stick to one db, and work around it's quirks?
I dunno, but on the other hand, it's quite pleasing to see how some of this bugs have disappeared during Sutekh's lifetime, which is always one of the strengths of FOSS.
Conceptually, the app is fairly simple, and I've been largely using it as a learning bed for pygtk. Consequently, it does things by hand, rather than the often recommended "use glade" approach. Personally, for something that now as a number of dynamic aspects to the gui (ability to rearrange panes, and so on), I think glade would have become quite painful, and straight pygtk coding is fairly easy once you wrap your head around the whole container paradigm. As an vehicle for learning pygtk, it has been pretty good.
But what has surprised me is it's ability to uncover database oddities. We have to date, discovered two issues with sqlobject (both fixed), had to work around sqlite's query optimiser by throwing in extra joins, triggered a segfault in sqlite (fixed in more recent sqlite's fortunately) and hit a bug in mysql query optimiser that makes certain queries run extremely slowly. WE ahven't triggered any postgresql bugs yet (although there have been a few weirdness's with certain versions of sqlobject talking to postgresql), but have had to fix several bugs in Sutekh's code because of postgres's much more stringent SQL implementation. For not a teribly complex database app, it's quite a impressive list.
So, do other people not trigger these bugs? Are we stretching corners of the databases that many other apps just don't hit (Sutekh has a number of tables, and can involve some quite complex joins)? Do other people just stick to one db, and work around it's quirks?
I dunno, but on the other hand, it's quite pleasing to see how some of this bugs have disappeared during Sutekh's lifetime, which is always one of the strengths of FOSS.
Subscribe to:
Posts (Atom)


