[Previous] New Community Website Project | Home | [Next] New Community Website: Marketing

Elliot Temple on December 6, 2020

Messages (30 of 218) (Show All Comments)

>> the idea was that all the other properties could be done through tags

> why? what problem does that solve?

The flexibility means ~arbitrarily complex stuff could be built on top. IDK what future requirements will be, but I think we'd be able to represent them with the current tagging system. That means easier, standardised implementation of features and reach of managment utils and less special case code. Also less risk of future incompatibility.

I hoped that it would be a good, principled, elegant foundation. (I don't think it's worked out like that, tho)

however, the conceptual downsides: we end up with SQL views for special cases, albiet somewhat modular; constraints are hard to enforce; the SQL queries are hard to reason about. we'd also probably end up wanting some UI customization around admin stuff, at least for constraint enforcement, but also consistency and ease of operation.

my method of tagging is basically built in to a decent graph db, tho. not just like superficially; it's the way graph dbs work. so that would mean simpler queries, better constraint mgmt, easier to reason about.

> deciding what specific things you need can keep them more organized than a generic system instead of just imagining it'll be able to do anything. we don't need the ability to have any number of arbitrary user permissions. we only need a few well-chosen permissions.

thought: if you have a flexible system it can still be worth making a specific system too. like we could have flags on a users record (e.g. is_sub, is_mod, is_admin), and by default an account has all set to false. if there's impl cost for building on the generic system (like permissions via tags -- which there certainly is) then it should be compared to the impl cost of the specific system. worst case you migrate from specific to generic when you need to.

I might try comparing performance of a 'permissions in user record' approach to see what the overhead of just my tag system is.

I was thinking today that queries on just the nodes should be super fast. like it's just an array of 25k nodes (in this case) tracking each other's position in the array -- not much overhead.


Max at 3:22 AM on January 20, 2021 | #192 | reply | quote

#191

> the concept of "it's super flexible" can be alluring.

the phrase "chasing universality" comes to mind.


Max at 3:23 AM on January 20, 2021 | #193 | reply | quote

#190 You could also have a nodes table and one table for each different type of relationship between nodes. E.g., you could have these tables for relationships: is_parent_of, links_to, references. Each of those tables would have two columns, source and target, each of which is a node ID.


jordancurve at 11:29 AM on January 20, 2021 | #194 | reply | quote

db/postgres: some surprising and good results

I don't quite know how to explain these results, but I've made an interesting development.

I refactored sql that I used in views to use Arel. In doing that I found a better way to do a particular join (and what I was joining with).

In SQLite the query takes about the same time: 2000ms (actually a bit worse than before).

nodes_admin_can_read_with_parent_root 2068.22 ms | result: 6

get_nodes_readable_by 2218.54 ms | result: 25019

But in postgres -- which used to be an order of magnitude worse at like 30000ms -- is now an order of magnitude better at around 300-500ms. That's including rails's overhead; the query itself is 200-300ms.

nodes_admin_can_read_with_parent_root 274.80 ms | result: 6

get_nodes_readable_by 452.95 ms | result: 25019

relevant code is here - the benchmark-db-fixes branch on fi-tools/cf-forum

You can run the benchmarks with

rails runner tools/benchmark.rb

and

rails runner -e devpg tools/benchmark.rb

(provided you have postgres set up)


Max at 7:23 AM on January 21, 2021 | #195 | reply | quote

some screenshots of the benchmarks with other results included:

image

image

(Note that just returning all children with no filtering is ~150ms for both)


Max at 7:28 AM on January 21, 2021 | #196 | reply | quote

I like the Ruby Weekly newsletter and read some stuff from today's issue:

https://rubyweekly.com/issues/536


curi at 11:44 AM on January 21, 2021 | #197 | reply | quote

https://github.com/hotwired/hotwire-rails

> Hotwire is an alternative approach to building modern web applications without using much JavaScript by sending HTML instead of JSON over the wire. This makes for fast first-load pages, keeps template rendering on the server, and allows for a simpler, more productive development experience in any programming language, without sacrificing any of the speed or responsiveness associated with a traditional single-page application.

> The heart of Hotwire is Turbo. A set of complimentary techniques for speeding up page changes and form submissions, dividing complex pages into components, and stream partial page updates over WebSocket. All without writing any JavaScript at all. And designed from the start to integrate perfectly with native hybrid applications for iOS and Android.

> While Turbo usually takes care of at least 80% of the interactivity that traditionally would have required JavaScript, there are still cases where a dash of custom code is required. Stimulus makes this easy with a HTML-centric approach to state and wiring.


curi at 11:48 AM on January 21, 2021 | #198 | reply | quote

https://www.learnhotwire.com

> Hotwire is a new approach to building applications you'd typically lean on React or Vue for - without all the Javascript. It's brought to you by the great minds behind Ruby on Rails, Basecamp, and Hey.com - so you know it puts developer productivity and happiness first.

https://github.com/hotwired/turbo

Note: Turbo was formerly called Turbolinks. There are search results under the old name, e.g.:

https://thoughtbot.com/upcase/videos/turbolinks

And Rails has Action Cable for WebSockets:

https://guides.rubyonrails.org/action_cable_overview.html


curi at 11:59 AM on January 21, 2021 | #199 | reply | quote

From the first email from the learnhotwire.com people:

> Hotwire helps us fulfill the dream most of us have as developers: building reactive, modern applications, quickly and easily. Frameworks like React and Vue are great, but it's not the code we love - it's the results. The truth is, we want to build our applications fast and not go crazy doing it. Unfortunately over the past few years, it feels like we have no choice but to use complex build chains, learn advance state management techniques, drown in an ocean of JSON responses, and write our code in multiple languages or dialects.

> ​

> We do all of this work, just to eventually render HTML in the browser.

> ​

> Hotwire frees us of all of this, allowing us to build the full experience in Ruby on Rails, avoiding the complex machinery, and delivery HTML directly to the browser, over the wire. We can still leverage the power of Javascript when we need it, but it is now a power tool, not something you use to build everything.

> ​

> Of course, most of these ideas aren't new, but rather, it leverages patterns that are battle tested in frameworks like Elixir's Phoenix (LiveView) and .NET (Blazor). There's even a pre-existing option for Ruby on Rails (Stimulus Reflex). However, Hotwire is created by the same experts, and with the same philosophies, as Ruby on Rails, Basecamp, and Hey.com.


curi at 12:04 PM on January 21, 2021 | #200 | reply | quote

#200 sounds pretty good. Will check it out tomorrow.


Max at 11:29 PM on January 21, 2021 | #201 | reply | quote

AGE / AgensGraph

> #182 maybe they renamed to Apache AGE? not clear on what's going on. this repo has recent updates https://github.com/bitnine-oss/AgensGraph-Extension

yup.

https://www.postgresql.org/about/news/announcing-age-a-multi-model-graph-database-extension-for-postgresql-2050/

> AGE, a multi-model graph database extension for PostgreSQL has been announced. **AGE is the successor to AgensGraph.** AGE will offer the same integration of SQL and Cypher without users having to discard their existing solutions, allow for a cleaner integration of AGE with PostgreSQL’s robust collection of other extensions, and expand scalability without sacrificing performance.

that page has a download link which goes to

https://github.com/bitnine-oss/AgensGraph-Extension

which then redirects you again to

https://github.com/apache/incubator-age

> The project is in alpha stage now and it is currently being developed in the form of an extension for PostgreSQL 11.

> The next Apache AGE release (0.3.0-incubating) will be available around Jan 15, 2021.

recent history:

> acdc110 11 hours ago

https://age.apache.org/

> AGE is currently being developed for the PostgreSQL 11 release and will support PostgreSQL 12 and 13 in 2021 and all the future releases of PostgreSQL.

under *Installing AGE* (no direct link)

> Docker images are available on Docker Hub and are based on the official PostgreSQL 11 Debian and Alpine images.

https://hub.docker.com/r/sorrell/agensgraph-extension

manual installation looks easy too. IDK if we lose anything by moving to postgres 11, but it's probs worth thinking about; and the right time to stdize build env if we're going to go with the current architecture. (on that note: we should probably figure out how to evaluate whether the architecture is good enough or not.)


Max at 3:49 AM on January 22, 2021 | #202 | reply | quote

rich text editor

Rails added a rich text editor. It's optional. At a glance it looks nice.

https://fullstackrubyonrails.com/blog/how-to-use-actiontext-in-rails-6

Demo and more info:

https://trix-editor.org

https://guides.rubyonrails.org/action_text_overview.html

It has nested quoting (click quote once on some text, then click the right indent button to add nesting levels).

But so far I haven't figured out what format it saves its data as, or what it can export to. So I don't know if it's markdown compatible or not (my initial thought is we want to save posts in markdown format in the db).


curi at 1:30 PM on January 22, 2021 | #203 | reply | quote

#203 I know it can output html, which we could potentially convert to markdown. That might require limiting the feature set. I don't know what the alternatives are and how good they are.


curi at 1:31 PM on January 22, 2021 | #204 | reply | quote

#168 I should amend what I wrote to say "transactions that write" instead of "writes". You can do a *bunch* of writes in a single transaction in SQLite and it'll be very fast. But if you need to update the database with a bunch of writes and have the results of each write be instantly available to other readers after each write completes, then that requires a lot of transactions and would be slower in SQLite than it would be in other databases.


jordancurve at 4:02 PM on January 24, 2021 | #205 | reply | quote

#204 I think html -> markdown will be harder than markdown -> html. I had a look around Trix docs but couldn't see anything advertising markdown support. There are other options tho, like some pure JS libraries for rendering markdown.

mb a good way to handle things: on the edit/create page we show raw markdown with a preview; the backend only sees markdown. then we can render markdown as HTML (user configurable?) for posts/comments etc. Adding buttons to do bold, etc, should be pretty straight forward even without a library (i.e. implementing it manually ourselves)


Max at 5:29 PM on January 24, 2021 | #206 | reply | quote

#202 it'd be nice and more user friendly if we had a rich text input option (preferably save as markdown) in addition to markdown input.


curi at 5:35 PM on January 24, 2021 | #207 | reply | quote

#207 think about tech illiterate people who repeatedly screw up quoting on this site or in email. markdown input with some buttons will confuse them and make the site a lot harder for them to use than FB or twitter. a WYSIWYG option will be easier for lots of people and is expected.


curi at 5:36 PM on January 24, 2021 | #208 | reply | quote

#206 I found this blogpost which goes through a few attempts to find a good markdown integration. https://www.codefellows.org/blog/how-to-create-a-markdown-friendly-blog-in-a-rails-app/

The author ends up going with the ruby library https://github.com/vmg/redcarpet (which looks active). apparently it was developed at github. looks like it has a decent feature-set and is configurable (so we could add extensions to do stuff like link comments like curi.us does). the rendering is done serverside, tho, so my guess is that we'd want to cache that. IDK tho; like maybe it'd make html-injection attacks easier. mb serverside rendering would be low overhead.


Max at 5:38 PM on January 24, 2021 | #209 | reply | quote

#206 i don't see much in the way of docs for trix but it looks nice so i skimmed some code:

https://github.com/basecamp/trix/blob/main/src/trix/models/document.coffee

some document export methods there:

toSerializableDocument

toString

toJSON

toConsole

and imports:

fromJSON

fromHTML

fromString

https://github.com/basecamp/trix/blob/main/src/trix/models/text.coffee

the text class has the same exports and the fromJSON import

and there's an html parser class

https://github.com/basecamp/trix/blob/main/src/trix/models/html_parser.coffee


curi at 5:43 PM on January 24, 2021 | #210 | reply | quote

#207

> it'd be nice and more user friendly if we had a rich text input option (preferably save as markdown) in addition to markdown input.

I think we can do that entirely client-side (in the UI) if we want. It can be the default.

Also WRT quoting I think there are win-win options like a feature that shows a 'quote' button when you select someone else's text that auto-inserts it into the reply field. it's win-win in the sense that it's useful for ppl who already know how to do good quoting.

> markdown input with some buttons will confuse them and make the site a lot harder for them to use than FB or twitter.

IDK, like reddit has pretty basic formatting stuff from memory. or at least WRT their old UI (can't remember wrt new UI).


Max at 5:45 PM on January 24, 2021 | #211 | reply | quote

#209 curi.us (and FI) uses redcarpet 2.3.0 for posts (but not comments). I chose redcarpet because I found a version that was compatible with old ruby. my research on this is some years old, but i recall there are a few major alternatives.

https://www.sitepoint.com/markdown-processing-ruby/

> We’re going to focus on 4 Ruby implementations of Markdown: kramdown, maruku, rdiscount, and redcarpet.

IIRC kramdown is what's used by my jekyll setup for https://www.elliottemple.com

if you wanted browser markdown rendering (good for e.g. live preview while writing a post, though i'd prefer a nice whole js editor someone already made) then you'd google for javascript markdown libraries not ruby ones.


curi at 5:49 PM on January 24, 2021 | #212 | reply | quote

> IDK, like reddit has pretty basic formatting stuff from memory. or at least WRT their old UI (can't remember wrt new UI).

most people who use reddit do not know how to use their text input stuff correctly, and a ton of people do not post or post less because they don't know how. reddit is incompetent and is massively harming their business by having an awful website (in dozens of ways). a tiny, self-selected group of reddit users do know how to use it and write quite a few of the comments, but a lot of people who write comments use few to no formatting features.


curi at 5:53 PM on January 24, 2021 | #213 | reply | quote

#209 that blogger is confused:

> Jekyll uses Liquid as a templating language, and I was using Rails’ out-of-the-box ERB. As far as I can tell, there is no way to make those two play nicely together.

you don't make templating languages play nicely with each other. but Rails supports using multiple templating languages in one project – just name files with different extensions. i don't know if there is any issue using Liquid files with Rails, but the issue is not about getting Liquid to play nice with ERB. also templating language and markdown to html conversion are separate issues.

btw there are some command line markdown to html tools, some of which may be pretty good, and they may also be relevant for exporting forum content as pdf, epub, mobi, etc. (cuz they'll do markdown -> html -> various. maybe some is done without the html in the middle, idk.) one of the tools that looks particularly promising is https://pandoc.org/index.html which has a ruby wrapper https://github.com/xwmx/pandoc-ruby


curi at 6:44 PM on January 24, 2021 | #214 | reply | quote

#214 re pandoc: yeah that's the 'anything -> anything' converter. we might lose stuff going from html -> markdown, tho.

pandoc is definitely useful for exporting.

personally I strongly prefer doing posts in markdown and just wrapping that on the client side. more flexible imo and can still have a rich WYSIWYG style editor. otherwise we entirely lose plaintext editing. plus, with a rich editor, users couldn't paste in markdown. IDK how they'd past stuff in without doing markdown->html on paste, anyway. Paste does support rich text but it'd need to be parsed. mb the rich editors support that already tho.

#213 yup, i didn't take into account selection bias there (regarding posts and posters). also my own bias of which subreddits i've browsed in the past.


Max at 6:51 PM on January 24, 2021 | #215 | reply | quote

#215 i don't understand what you're arguing with. i wasn't talking about not having markdown input support. i want markdown as an input. i also want a user friendly input option.


curi at 6:55 PM on January 24, 2021 | #216 | reply | quote

rails recursive includes

This isn't general, but i did find a way to do 'recursive' includes. One reason I did all the arel stuff (see models/node.rb) was b/c doing this sort of thing wasn't obvious. Note: it's not as good as the recursive method IMO, and doesn't consider permissions:

Node.includes(direct_children: [:direct_children, { direct_children: [:direct_children] }])

The debut output for SQL queries looks like:

Node Load (1.6ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."id" = $1 LIMIT $2 [["id", 0], ["LIMIT", 1]]

Node Load (1.0ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."parent_id" = $1 [["parent_id", 0]]

Node Load (1.2ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."parent_id" IN ($1, $2, $3, $4, $5, $6) [[nil, 2], [nil, 3], [nil, 4], [nil, 1], [nil, 14], [nil, 18]]

Node Load (1.1ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."parent_id" IN ($1, $2, $3, $4, $5, $6) [[nil, 5], [nil, 6], [nil, 15], [nil, 20], [nil, 21], [nil, 19]]


Max at 7:48 PM on January 25, 2021 | #217 | reply | quote

cf forum rails db progress

done a bunch of work on using db views and caching some stuff we can easily calculate incrementally.

using Node.insert_all! the seeds file can generate 88k nodes (forum posts) in under 3 minutes - fast enough to be useful.

using materialized views: all the permissions stuff is cached and refreshes are fast - 400ms with concurrently=false (blocks reads) and 800ms with concurrently=true.

rendering is fast: although there's some n+1 ish stuff going on, viewing a topic with 30k descendants takes less than 200ms and the SQL queries take like 4ms each. Note: this is capped at a depth of 3, and the tree has branching factor 3, so 40 nodes or so.

When I set max_depth = 4 (so ~120 nodes), this is the split between active record and views:

Completed 200 OK in 558ms (Views: 411.1ms | ActiveRecord: 139.1ms | Allocations: 248131)

with max_depth=3, some stats:


Max at 2:34 AM on January 27, 2021 | #218 | reply | quote

db progress: all pages rendering in <100ms with permissions.

some pictures / profile stats: https://github.com/fi-tools/cf-forum/pull/23#issuecomment-768514477

one thing I forgot to mention with #218 is that it's a very rails-y solution, too:

image


Max at 11:20 AM on January 27, 2021 | #219 | reply | quote

todo lists (that u can check off) and calendar stuff (e.g. notify at certain time, or recurring events like being asked a question on a schedule) are possible features.


curi at 11:59 AM on February 13, 2021 | #221 | reply | quote

(This is an unmoderated discussion forum. Discussion info.)