Why are partial PostgreSQL HASH indices not smaller than full indices?Why does MySQL not have hash indices on MyISAM or InnoDB?Adding index to large mysql tablesWhy would function based indices I’ve created lower the cost but not show up in the explain plan breakdown?Optimizing indexesPostgreSQL indices on multi fields queriesPostgreSQL not using partial index when using boolean in WHERE clause512 Bytes are not being used from SQL Server's 8 KByte data pageIndexing strategy for VARCHAR2 LIKE searchHow is it possible for Hash Index not to be faster than Btree for equality lookups?PostgreSQL suitabilty of Hash Index on PK and FK

Why does a car's steering wheel get lighter with increasing speed

Why do we say 'Pairwise Disjoint', rather than 'Disjoint'?

When Central Limit Theorem breaks down

Why isn't P and P/poly trivially the same?

Why restrict private health insurance?

What does *dead* mean in *What do you mean, dead?*?

How does a sound wave propagate?

Why do we call complex numbers “numbers” but we don’t consider 2-vectors numbers?

Are brahmins allowed to drink alcohol?

Vector-transposing function

What is the orbit and expected lifetime of Crew Dragon trunk?

Can I challenge the interviewer to give me a proper technical feedback?

What is the oldest European royal house?

How do you make a gun that shoots melee weapons and/or swords?

How to educate team mate to take screenshots for bugs with out unwanted stuff

Does an unused member variable take up memory?

How to distinguish easily different soldier of ww2?

The (Easy) Road to Code

How to install "rounded" brake pads

Mixed Feelings - What am I

What is the purpose of a disclaimer like "this is not legal advice"?

Is divide-by-zero a security vulnerability?

Will the concrete slab in a partially heated shed conduct a lot of heat to the unconditioned area?

Is there a math expression equivalent to the conditional ternary operator?



Why are partial PostgreSQL HASH indices not smaller than full indices?


Why does MySQL not have hash indices on MyISAM or InnoDB?Adding index to large mysql tablesWhy would function based indices I’ve created lower the cost but not show up in the explain plan breakdown?Optimizing indexesPostgreSQL indices on multi fields queriesPostgreSQL not using partial index when using boolean in WHERE clause512 Bytes are not being used from SQL Server's 8 KByte data pageIndexing strategy for VARCHAR2 LIKE searchHow is it possible for Hash Index not to be faster than Btree for equality lookups?PostgreSQL suitabilty of Hash Index on PK and FK













4















I want to create the most efficient index for a sparsely populated column. I only need equality operations, so a HASH index should be beneficial.



Now I'm wondering why a partial HASH index isn't smaller than a full hash index:



CREATE INDEX full_hash ON mytable USING HASH(my_id); # 256 MB
CREATE INDEX partial_hash ON mytable USING HASH(my_id) WHERE my_ID IS NOT NULL; # 256 MB

CREATE INDEX full_btree ON mytable (my_id); # 537 MB
CREATE INDEX partial_btree ON mytable (my_id) WHERE my_ID IS NOT NULL; # 32 MB


Both hash indices take exactly the same amount of space (as shown in pgHero). However, when using standard BTREE indices, the partial index takes only 5% of the space of the full index.



Are partial HASH indices not supported in PostgreSQL 10?










share|improve this question


























    4















    I want to create the most efficient index for a sparsely populated column. I only need equality operations, so a HASH index should be beneficial.



    Now I'm wondering why a partial HASH index isn't smaller than a full hash index:



    CREATE INDEX full_hash ON mytable USING HASH(my_id); # 256 MB
    CREATE INDEX partial_hash ON mytable USING HASH(my_id) WHERE my_ID IS NOT NULL; # 256 MB

    CREATE INDEX full_btree ON mytable (my_id); # 537 MB
    CREATE INDEX partial_btree ON mytable (my_id) WHERE my_ID IS NOT NULL; # 32 MB


    Both hash indices take exactly the same amount of space (as shown in pgHero). However, when using standard BTREE indices, the partial index takes only 5% of the space of the full index.



    Are partial HASH indices not supported in PostgreSQL 10?










    share|improve this question
























      4












      4








      4








      I want to create the most efficient index for a sparsely populated column. I only need equality operations, so a HASH index should be beneficial.



      Now I'm wondering why a partial HASH index isn't smaller than a full hash index:



      CREATE INDEX full_hash ON mytable USING HASH(my_id); # 256 MB
      CREATE INDEX partial_hash ON mytable USING HASH(my_id) WHERE my_ID IS NOT NULL; # 256 MB

      CREATE INDEX full_btree ON mytable (my_id); # 537 MB
      CREATE INDEX partial_btree ON mytable (my_id) WHERE my_ID IS NOT NULL; # 32 MB


      Both hash indices take exactly the same amount of space (as shown in pgHero). However, when using standard BTREE indices, the partial index takes only 5% of the space of the full index.



      Are partial HASH indices not supported in PostgreSQL 10?










      share|improve this question














      I want to create the most efficient index for a sparsely populated column. I only need equality operations, so a HASH index should be beneficial.



      Now I'm wondering why a partial HASH index isn't smaller than a full hash index:



      CREATE INDEX full_hash ON mytable USING HASH(my_id); # 256 MB
      CREATE INDEX partial_hash ON mytable USING HASH(my_id) WHERE my_ID IS NOT NULL; # 256 MB

      CREATE INDEX full_btree ON mytable (my_id); # 537 MB
      CREATE INDEX partial_btree ON mytable (my_id) WHERE my_ID IS NOT NULL; # 32 MB


      Both hash indices take exactly the same amount of space (as shown in pgHero). However, when using standard BTREE indices, the partial index takes only 5% of the space of the full index.



      Are partial HASH indices not supported in PostgreSQL 10?







      postgresql index index-tuning postgresql-10






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 18 hours ago









      Ortwin GentzOrtwin Gentz

      1324




      1324




















          2 Answers
          2






          active

          oldest

          votes


















          5














          I would argue that this is a bug in the hash index code. When you create an index on an already-populated table, it tries to pre-size the index to hold all the data so that it doesn't have to keep splitting buckets as the index is created. But the code for doing this does not take the NULL fraction of the column nor (apparently) the selectivity of the partial index clause into account, so it arrives at a too-large number for the pre-sizing.



          If you were to create the index first, and then populated the table, you will find that the hash index is small, whether you made it partial or not. If the table is going to grow substantially after the index is created, the extra space consumed by the index upon original creation will be put to good use.






          share|improve this answer


















          • 3





            I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

            – jjanes
            10 hours ago






          • 1





            Oh, and I submitted a bug already: postgresql.org/message-id/…

            – Ortwin Gentz
            9 hours ago


















          4














          It's not explicitly stated in the documentation, but in the source code there is the following comment:



          /*
          * We do not insert null values into hash indexes. This is okay because
          * the only supported search operator is '=', and we assume it is strict.
          */


          So the is not null predicate does indeed change nothing, as null values are always ignored for hash indexes (which does make sense, as comparing null values with = would never return true).






          share|improve this answer




















          • 2





            Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

            – Ortwin Gentz
            17 hours ago











          • This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

            – jjanes
            16 hours ago











          • The full btree index is more than double the size of the hash index.

            – Ortwin Gentz
            11 hours ago










          Your Answer








          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "182"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f231647%2fwhy-are-partial-postgresql-hash-indices-not-smaller-than-full-indices%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          5














          I would argue that this is a bug in the hash index code. When you create an index on an already-populated table, it tries to pre-size the index to hold all the data so that it doesn't have to keep splitting buckets as the index is created. But the code for doing this does not take the NULL fraction of the column nor (apparently) the selectivity of the partial index clause into account, so it arrives at a too-large number for the pre-sizing.



          If you were to create the index first, and then populated the table, you will find that the hash index is small, whether you made it partial or not. If the table is going to grow substantially after the index is created, the extra space consumed by the index upon original creation will be put to good use.






          share|improve this answer


















          • 3





            I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

            – jjanes
            10 hours ago






          • 1





            Oh, and I submitted a bug already: postgresql.org/message-id/…

            – Ortwin Gentz
            9 hours ago















          5














          I would argue that this is a bug in the hash index code. When you create an index on an already-populated table, it tries to pre-size the index to hold all the data so that it doesn't have to keep splitting buckets as the index is created. But the code for doing this does not take the NULL fraction of the column nor (apparently) the selectivity of the partial index clause into account, so it arrives at a too-large number for the pre-sizing.



          If you were to create the index first, and then populated the table, you will find that the hash index is small, whether you made it partial or not. If the table is going to grow substantially after the index is created, the extra space consumed by the index upon original creation will be put to good use.






          share|improve this answer


















          • 3





            I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

            – jjanes
            10 hours ago






          • 1





            Oh, and I submitted a bug already: postgresql.org/message-id/…

            – Ortwin Gentz
            9 hours ago













          5












          5








          5







          I would argue that this is a bug in the hash index code. When you create an index on an already-populated table, it tries to pre-size the index to hold all the data so that it doesn't have to keep splitting buckets as the index is created. But the code for doing this does not take the NULL fraction of the column nor (apparently) the selectivity of the partial index clause into account, so it arrives at a too-large number for the pre-sizing.



          If you were to create the index first, and then populated the table, you will find that the hash index is small, whether you made it partial or not. If the table is going to grow substantially after the index is created, the extra space consumed by the index upon original creation will be put to good use.






          share|improve this answer













          I would argue that this is a bug in the hash index code. When you create an index on an already-populated table, it tries to pre-size the index to hold all the data so that it doesn't have to keep splitting buckets as the index is created. But the code for doing this does not take the NULL fraction of the column nor (apparently) the selectivity of the partial index clause into account, so it arrives at a too-large number for the pre-sizing.



          If you were to create the index first, and then populated the table, you will find that the hash index is small, whether you made it partial or not. If the table is going to grow substantially after the index is created, the extra space consumed by the index upon original creation will be put to good use.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered 16 hours ago









          jjanesjjanes

          13.6k917




          13.6k917







          • 3





            I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

            – jjanes
            10 hours ago






          • 1





            Oh, and I submitted a bug already: postgresql.org/message-id/…

            – Ortwin Gentz
            9 hours ago












          • 3





            I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

            – jjanes
            10 hours ago






          • 1





            Oh, and I submitted a bug already: postgresql.org/message-id/…

            – Ortwin Gentz
            9 hours ago







          3




          3





          I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

          – jjanes
          10 hours ago





          I've started a thread about this on the developers mailing list (postgresql.org/message-id/flat/…) if anyone here would like to follow it.

          – jjanes
          10 hours ago




          1




          1





          Oh, and I submitted a bug already: postgresql.org/message-id/…

          – Ortwin Gentz
          9 hours ago





          Oh, and I submitted a bug already: postgresql.org/message-id/…

          – Ortwin Gentz
          9 hours ago













          4














          It's not explicitly stated in the documentation, but in the source code there is the following comment:



          /*
          * We do not insert null values into hash indexes. This is okay because
          * the only supported search operator is '=', and we assume it is strict.
          */


          So the is not null predicate does indeed change nothing, as null values are always ignored for hash indexes (which does make sense, as comparing null values with = would never return true).






          share|improve this answer




















          • 2





            Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

            – Ortwin Gentz
            17 hours ago











          • This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

            – jjanes
            16 hours ago











          • The full btree index is more than double the size of the hash index.

            – Ortwin Gentz
            11 hours ago















          4














          It's not explicitly stated in the documentation, but in the source code there is the following comment:



          /*
          * We do not insert null values into hash indexes. This is okay because
          * the only supported search operator is '=', and we assume it is strict.
          */


          So the is not null predicate does indeed change nothing, as null values are always ignored for hash indexes (which does make sense, as comparing null values with = would never return true).






          share|improve this answer




















          • 2





            Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

            – Ortwin Gentz
            17 hours ago











          • This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

            – jjanes
            16 hours ago











          • The full btree index is more than double the size of the hash index.

            – Ortwin Gentz
            11 hours ago













          4












          4








          4







          It's not explicitly stated in the documentation, but in the source code there is the following comment:



          /*
          * We do not insert null values into hash indexes. This is okay because
          * the only supported search operator is '=', and we assume it is strict.
          */


          So the is not null predicate does indeed change nothing, as null values are always ignored for hash indexes (which does make sense, as comparing null values with = would never return true).






          share|improve this answer















          It's not explicitly stated in the documentation, but in the source code there is the following comment:



          /*
          * We do not insert null values into hash indexes. This is okay because
          * the only supported search operator is '=', and we assume it is strict.
          */


          So the is not null predicate does indeed change nothing, as null values are always ignored for hash indexes (which does make sense, as comparing null values with = would never return true).







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited 17 hours ago

























          answered 18 hours ago









          a_horse_with_no_namea_horse_with_no_name

          40.5k777113




          40.5k777113







          • 2





            Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

            – Ortwin Gentz
            17 hours ago











          • This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

            – jjanes
            16 hours ago











          • The full btree index is more than double the size of the hash index.

            – Ortwin Gentz
            11 hours ago












          • 2





            Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

            – Ortwin Gentz
            17 hours ago











          • This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

            – jjanes
            16 hours ago











          • The full btree index is more than double the size of the hash index.

            – Ortwin Gentz
            11 hours ago







          2




          2





          Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

          – Ortwin Gentz
          17 hours ago





          Interesting. So apparently, hash indexes aren't appropriate for sparsely populated columns. I tested with a column even less populated (only a few 100 records out of >10 m total) and the index took 256 MB as well. So it looks like the space of a hash index only depends on table size, not on the number of different indexable values.

          – Ortwin Gentz
          17 hours ago













          This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

          – jjanes
          16 hours ago





          This explains why the two HASH indexes are the same size as each other, but not why they are so large compared to the btree indexes.

          – jjanes
          16 hours ago













          The full btree index is more than double the size of the hash index.

          – Ortwin Gentz
          11 hours ago





          The full btree index is more than double the size of the hash index.

          – Ortwin Gentz
          11 hours ago

















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Database Administrators Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f231647%2fwhy-are-partial-postgresql-hash-indices-not-smaller-than-full-indices%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Möglingen Índice Localización Historia Demografía Referencias Enlaces externos Menú de navegación48°53′18″N 9°07′45″E / 48.888333333333, 9.129166666666748°53′18″N 9°07′45″E / 48.888333333333, 9.1291666666667Sitio web oficial Mapa de Möglingen«Gemeinden in Deutschland nach Fläche, Bevölkerung und Postleitzahl am 30.09.2016»Möglingen

          Virtualbox - Configuration error: Querying “UUID” failed (VERR_CFGM_VALUE_NOT_FOUND)“VERR_SUPLIB_WORLD_WRITABLE” error when trying to installing OS in virtualboxVirtual Box Kernel errorFailed to open a seesion for the virtual machineFailed to open a session for the virtual machineUbuntu 14.04 LTS Virtualbox errorcan't use VM VirtualBoxusing virtualboxI can't run Linux-64 Bit on VirtualBoxUnable to insert the virtual optical disk (VBoxguestaddition) in virtual machine for ubuntu server in win 10VirtuaBox in Ubuntu 18.04 Issues with Win10.ISO Installation

          Torre de la Isleta Índice Véase también Referencias Bibliografía Enlaces externos Menú de navegación38°25′58″N 0°23′02″O / 38.43277778, -0.3838888938°25′58″N 0°23′02″O / 38.43277778, -0.38388889Torre de la Illeta de l’Horta o Torre Saleta. Base de datos de bienes inmuebles. Patrimonio Cultural. Secretaría de Estado de CulturaFicha BIC Torre de la Illeta de l’Horta. Dirección General de Patrimonio Cultural. Generalitat ValencianaLugares de interés. Ayuntamiento del CampelloTorre de la Isleta en CastillosNet.org