DISQUS

DISQUS Hello! Homo-Adminus Blog is using DISQUS, a powerful comment system, to manage its comments. Learn more.

Community Page

Jump to original thread »
Author

Using Sphinx for Non-Fulltext Queries

Started by Scoundrel · 9 months ago

How often do you think about the reasons why your favorite RDBMS sucks? Last few months I was doing this quite often and yes, my favorite RDBMS is MySQL. The reason why I was thinking so because one of my recent tasks at Scribd was fixing scalability problems in documents browsing.
The probl ... Continue reading »

10 comments

  • We use this technique for filtering searches on categories, price, etc. We just stick cat_39 in a deal and add that to the sphinx query. Works like a charm.
  • Brian,

    There are two different techniques - one is adding pseudo keywords and other is using filters and "Full Table Scan" query.

    Pseudo keywords works if there are few of them in the query and they are well selective.

    If you will have 10 keywords each with 90% selectivity this would not work well.

    Filters w full table scan however can be slower with good selectivity but you can have a lot of non selective filters with very low penalty.

    Also note Sphinx "full table scan" has optimizations so it often can skip a lot of "blocks" based on filters which makes it kind of similar to indexed lookup
  • Good блог at you, cognitive:)
  • Do you guys use Sphinx to index the contents of your documents, or only metadata about them that is stored in the database?
  • 2W. Andrew Loe III: No, for browse index we use documents' attributes only.
  • Hi!

    First of all, you blog is very interesting and I like it, salute from Ireland to you! ;)

    The problem is with MySQL(like with every DBMS written by folks wo/ corporate experience) still the relative small amount of features to handle something bigger than the average.

    Let me show you what i am thinking about:

    http://www.ibm.com/developerworks/db2/library/t...

    http://www.oracle.com/technology/products/oracl...

    http://msdn.microsoft.com/en-us/library/ms34514...

    And just for the record, innodb is an Oracle product.

    Maybe you can try this:
    http://dev.mysql.com/doc/refman/5.1/en/partitio...

    Regards,
    lix
  • Hello Alexei,
    I have one comment regarding this:
    From what I've read so far Sphynx is great only for non-constantly-changing-data... Am I correct.
    If you have thousands of writes per second and you want the info to be available within 1-3 second span how would you do that with sphynx??? if reindexing takes 20-40 seconds of 3Mil rows as you said...
  • Andrew has said that online add and remove is coming in 0.9.9. When will it be ready? Don't know. But that will be a big, big jump in usefulness.
  • Yes I agree sphinx makes for a great general purpose index. We put a huge database in sphinx for full text, but realised it could be used an index on many of the other fields too!


    @Pavel

    Sphinx can search multiple indexes in one query, so the basic idea is to create a main index, and the a delta index, you just more regually reindex the delta index. You can even have multiple delta indexes - basically constantly reindexing the changes. You can also use line attribute updates to 'delete' items from the main index without reindexing.
  • Thanks for the info. May God have mercy on us all.

Add New Comment

Returning? Login